[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

ext3 bug?



Hi,
I have a severe filesystem crash on a machine with a single ext3 filesystem.
It is the second time that this happens to me (on two different machines that
are identical in hardware setup and mostly in software - I cloned them when
I started to work).

It looks like the journal (or something else, i am not an expert here) has
gone wild and overwritten loads of inodes. I have a single filesys for every-
thing (I know, a bad habit), which has been remounted read-only. Using a ssh
tunnel that is still open, I had a look at the fs and found this:

root wally /]# ls -l sbin/
ls: sbin/ipfwadm: Eingabe-/Ausgabefehler
ls: sbin/depmod.modutils: Eingabe-/Ausgabefehler
ls: sbin/update-modules.modutils: Eingabe-/Ausgabefehler
ls: sbin/modprobe.Lmodutils: Eingabe-/Ausgabefehler
ls: sbin/mount.smbfs: Eingabe-/Ausgabefehler
ls: sbin/mount.smb: Eingabe-/Ausgabefehler
insgesamt 1075231521
-rwxr-xr-x    1 root     root        14408 22. M� 2002  badblocks
-rwxr-xr-x    1 root     root         7360 31. Mai 2003  blockdev
-rwxr-xr-x    1 root     root        47944 31. Mai 2003  cfdisk
[... normal entries salvaged]
-rwxr-xr-x    1 root     root          413 29. Mai 2002  fsck.nfs
lrwxrwxrwx    1 root     root            7 20. Feb 2003  fsck.vfat -> dosfsck
?rwxrw-rwt  13639 21845    18754          23 27. J� 1994  genksyms
-rwxr-xr-x    1 root     root        14184 31. Mai 2003  getty
-rwxr-xr-x    1 root     root       121032 16. Apr 2003  grub
-rwxr-xr-x    1 root     root         2944 16. Apr 2003  grub-floppy
-rwxr-xr-x    1 root     root        12110 16. Apr 2003  grub-install
-rwxr-xr-x    1 root     root         2301 16. Apr 2003  grub-md5-crypt
-rwxr-xr-x    1 root     root         2473 16. Apr 2003  grub-terminfo
-rwxr-xr-x    1 root     root         9104 29. Mai 2002  halt
?-----xr-x  2240 1931476992 1952802008 1700749935  1. J� 1970  hciattach
cr-Srwsr-T  2240 487270   25816      5,  16  1. J� 1970  hciconfig
-rwxr-xr-x    1 root     root        27048 10. Apr 2003  hcid
[...]
-rwxr-xr-x    1 root     root        14092 24. Nov 2001  iptunnel
lrwxrwxrwx    1 root     root           15 23. Jul 08:53 kallsyms -> insmod.modutils
--wsrwxrw-  16406 3257942038 1075218782 4618131408204237070 27. J� 2004  kallsyms.modutils
-rwxr-xr-x    1 root     root         5776 12. Nov 2001  kbdrate
?rwxrw-rwt  16896 21845    12079          26 27. J� 1994  kernelversion
-rwxr-xr-x    1 root     root         8948 29. Mai 2002  killall5
-rwxr-xr-x    1 root     root        19672  3. J� 2002  klogd
lrwxrwxrwx    1 root     root           15 23. Jul 08:53 ksyms -> insmod.modutils
sr-Srw----  16407 3423617046 1075238984 1075656696 27. J� 2004  ksyms.modutils
-rwxr-xr-x    1 root     root       430636  8. Apr 2003  ldconfig
-rwxr-xr-x    1 root     root        20600 31. Mai 2003  losetup
lrwxrwxrwx    1 root     root           10 23. Jul 08:53 lsmod -> /bin/lsmod
?-wxrwSrwx  2063 21845    36537          73  1. J� 1970  lsmod.Lmodutils
s--x--S---  16406 131072   10376          13  1. J� 1970  lsmod.modutils
--wxrwsrwT  16406 2903523350 1074366302 4614300640973588238 27. J� 2004  lspci
-rwxr-xr-x    1 root     root        43485  3. M� 2002  MAKEDEV
-rwxr-xr-x    1 root     root        10188 24. Nov 2001  mii-tool
[...]
lrwxrwxrwx    1 root     root            7 20. Feb 2003  mkfs.msdos -> mkdosfs
lrwxrwxrwx    1 root     root            7 20. Feb 2003  mkfs.vfat -> mkdosfs
?-ws------  16407 85999638 1075773488 1075233008 28. J� 2004  mkswap
-rwxr-xr-x    1 root     root         8752 10. Jul 2003  modinfo
-rwxr-xr-x    1 root     root        39912 10. Apr 2003  modinfo.modutils
-rwxr-xr-x    1 root     root        19796 10. Jul 2003  modprobe
lrwx---rwT  2053 2209351685 1074963494 134587350 23. J� 2004  modprobe.Lmodutils
lrwxrwxrwx    1 root     root           15  4. Apr 2003  modprobe.modutils -> insmod.modutils
-rwxr-xr-x    1 root     root         7108 24. Nov 2001  nameif
-rwxr-xr-x    1 root     root         2808 31. Mai 2003  pivot_root
-rwxr-xr-x    1 root     root         4716 24. Nov 2001  plipconfig
-rwxr-xr-x    1 root     root         3324 18. M� 2001  pmap_dump
-rwxr-xr-x    1 root     root         3372 18. M� 2001  pmap_set
-rwxr-xr-x    1 root     root        11692 18. M� 2001  portmap
?-ws--s--T  16409 694698007 1075434216 1075242124 31. J� 2004  poweroff
-rwxr-xr-x    1 root     root        18860 24. Nov 2001  rarp
-rwxr-xr-x    1 root     root         5088 31. Mai 2003  raw
?rwSrwx--T  16406 113786907 1075297776 1075751024 27. J� 2004  reboot
-rwxr-xr-x    1 root     root        19056 22. M� 2002  resize2fs
-rwxr-xr-x    1 root     root         7812 10. Jul 2003  rmmod
lrwxrwxrwx    1 root     root            6 16. Apr 2003  rmmod.Lmodutils -> insmod
lrwxrwxrwx    1 root     root           15  4. Apr 2003  rmmod.modutils -> insmod.modutils
?--xr----T  16408 276316184 1075252624 1075773640 27. J� 2004  rmt
-rwxr-xr-x    1 root     root        42060 24. Nov 2001  route
-rwxr-xr-x    1 root     root         3196  9. Jul 2003  rpc.lockd
[..]
-rwxr-xr-x    1 root     root       153792  2. Apr 2002  tc
?rws--S---  16413 3167764512 1075212976 1075238948  2. Feb 2004  telinit
-rwxr-xr-x    1 root     root        22312 22. M� 2002  tune2fs
-rwsr-xr-x    1 root     root        14508 21. J� 2002  unix_chkpwd
-rwxr-xr-x    1 root     root        16560 16. Apr 2003  update-grub
-rwxr-xr-x    1 root     root         2807 10. Jul 2003  update-modules

This can be found throughout the fs - /lib, /usr/bin, /etc and others all are a mess.
Luckily I could get hold of the database files and the web content that my
students were working on...

I suspect the reason for the crash to correspond with large files: I had a
2.6 Gb file in /tmp (a gzipped partition) that I copied over to another machine, and
that night it crashed... It ran fine the last year, though.

A similar incident happened to my first machine, where I moved a hd image to an
external firewire disk. After deleting it from my machine, I was stranded with a
totally broken system that I could only get up again with a boot cd, and after an
fs check I now still have 16000 files in /lost+found (around 1.5 gig of data...)

The hardware on the machine looks like this:
00:00.0 Host bridge: Intel Corp. 82850 850 (Tehama) Chipset Host Bridge (MCH) (rev 04)
00:01.0 PCI bridge: Intel Corp. 82850 850 (Tehama) Chipset AGP Bridge (rev 04)
00:1e.0 PCI bridge: Intel Corp. 82820 820 (Camino 2) Chipset PCI (rev 04)
00:1f.0 ISA bridge: Intel Corp. 82820 820 (Camino 2) Chipset ISA Bridge (ICH2) (rev 04)
00:1f.1 IDE interface: Intel Corp. 82820 820 (Camino 2) Chipset IDE U100 (rev 04)
00:1f.2 USB Controller: Intel Corp. 82820 820 (Camino 2) Chipset USB (Hub A) (rev 04)
00:1f.3 SMBus: Intel Corp. 82820 820 (Camino 2) Chipset SMBus (rev 04)
00:1f.4 USB Controller: Intel Corp. 82820 820 (Camino 2) Chipset USB (Hub B) (rev 04)
00:1f.5 Multimedia audio controller: Intel Corp. 82820 820 (Camino 2) Chipset AC'97 Audio Controller (rev 04)
01:00.0 VGA compatible controller: Matrox Graphics, Inc.: Unknown device 0527 (rev 03)
02:04.0 USB Controller: NEC Corporation USB (rev 41)
02:04.1 USB Controller: NEC Corporation USB (rev 41)
02:04.2 USB Controller: NEC Corporation: Unknown device 00e0 (rev 02)
02:0b.0 Ethernet controller: 3Com Corporation 3c905C-TX [Fast Etherlink] (rev 78)
02:0c.0 FireWire (IEEE 1394): VIA Technologies, Inc. OHCI Compliant IEEE 1394 Host Controller (rev 46)
[vogl /net/home2/vogl]$ cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 15
model : 2
model name : Intel(R) Pentium(R) 4 CPU 2.40GHz
stepping : 7
cpu MHz : 2405.505
cache size : 512 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe cid
bogomips : 4797.23
[vogl /net/home2/vogl]$ cat /proc/interrupts
CPU0
0: 23797604 IO-APIC-edge timer
1: 126073 IO-APIC-edge keyboard
2: 0 XT-PIC cascade
4: 2 IO-APIC-edge serial
8: 4 IO-APIC-edge rtc
14: 355574 IO-APIC-edge ide0
15: 5 IO-APIC-edge ide1
16: 0 IO-APIC-level Matrox Graphics, Inc. MGA Parhelia AGP
17: 3984 IO-APIC-level Intel 82801BA-ICH2
19: 18062239 IO-APIC-level usb-uhci
20: 169 IO-APIC-level ohci1394
22: 0 IO-APIC-level acpi
23: 10021486 IO-APIC-level eth0, ehci_hcd, usb-uhci
NMI: 0
LOC: 23796986
ERR: 0
MIS: 0


[root wally /]# uptime
 10:02:55 up 23 days, 23:48,  4 users,  load average: 0.00, 0.00, 0.00
[root wally /]# uname -a
Linux wally 2.4.23 #1 Tue Dec 23 09:44:22 CET 2003 i686 unknown
[root wally /]# mount
/dev/ide/host0/bus0/target0/lun0/part1 on / type ext3 (rw,errors=remount-ro)
proc on /proc type proc (rw)
devpts on /dev/pts type devpts (rw)
/dev/ide/host0/bus0/target0/lun0/part5 on /media type vfat (rw,noexec,nodev,gid=2000,umask=002)
usbdevfs on /proc/bus/usb type usbdevfs (rw)
automount(pid1324) on /data/net type autofs (rw,fd=5,pgrp=1324,minproto=2,maxproto=4)
automount(pid1348) on /net type autofs (rw,fd=5,pgrp=1348,minproto=2,maxproto=4)
automount(pid1319) on /var/autofs/misc type autofs (rw,fd=5,pgrp=1319,minproto=2,maxproto=4)


Anything I can do before formatting the disk to lighten things up? Simon


-- _______________________________________________________________________ Dr. Simon Vogl Institut für Pervasive Computing, Johannes Kepler Universität Linz Altenberger Straße 69, A-4040 Linz, Austria

Tel: +43 732 2468-8517, Fax: +43 732 2468-8426
mailto: vogl soft uni-linz ac at,  http://www.soft.uni-linz.ac.at/









[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]