[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [Linux-cluster] Any thoughts on losing mount?



> The error message indicates resource group (RG) may get corrupted. Have
> you tried to do an fsck (or did it fixes anything) ?

Should this be while the partition is unmapped on any of the nodes?

# ./fsck /dev/mapper/VolGroup03-web
fsck 1.35 (28-Feb-2004)
e2fsck 1.35 (28-Feb-2004)
Couldn't find ext2 superblock, trying backup blocks...
fsck.ext2: Bad magic number in super-block while trying to open 
/dev/mapper/VolGroup03-web

I've also seen this in the log;

compdev kernel: GFS: Trying to join cluster "lock_dlm", "vgcomp:qm"
compdev kernel: GFS: fsid=vgcomp:qm.1: Joined cluster. Now mounting FS...
compdev kernel: GFS: fsid=vgcomp:qm.1: jid=1: Trying to acquire journal 
lock...
compdev kernel: GFS: fsid=vgcomp:qm.1: jid=1: Looking at journal...
compdev kernel: GFS: fsid=vgcomp:qm.1: jid=1: Done
compdev kernel: GFS: fsid=vgcomp:web.3: fatal: filesystem consistency error
compdev kernel: GFS: fsid=vgcomp:web.3:   RG = 31104599
compdev kernel: GFS: fsid=vgcomp:web.3:   function = gfs_setbit
compdev kernel: GFS: fsid=vgcomp:web.3:   file = 
/home/xos/gen/updates-2007-11/xlrpm29472/rpm/BUILD/gfs-
kernel-2.6.9-72/up/src/gfs/bits.c, line = 71
compdev kernel: GFS: fsid=vgcomp:web.3:   time = 1196105648
compdev kernel: GFS: fsid=vgcomp:web.3: about to withdraw from the cluster
compdev kernel: GFS: fsid=vgcomp:web.3: waiting for outstanding I/O
compdev kernel: GFS: fsid=vgcomp:web.3: telling LM to withdraw
compdev kernel: lock_dlm: withdraw abandoned memory
compdev kernel: GFS: fsid=vgcomp:web.3: withdrawn

and;

compdev kernel: GFS: fsid=vgcomp:web.3: Scanning for log elements...
compdev kernel: GFS: fsid=vgcomp:web.3: Found 1 unlinked inodes
compdev kernel: GFS: fsid=vgcomp:web.3: Found quota changes for 0 IDs
compdev kernel: GFS: fsid=vgcomp:web.3: Done
compdev kernel: GFS: fsid=vgcomp:web.3: fatal: filesystem consistency error
compdev kernel: GFS: fsid=vgcomp:web.3:   RG = 31104599
compdev kernel: GFS: fsid=vgcomp:web.3:   function = gfs_setbit
compdev kernel: GFS: fsid=vgcomp:web.3:   file = 
/home/xos/gen/updates-2007-11/xlrpm29472/rpm/BUILD/gfs-
kernel-2.6.9-72/up/src/gfs/bits.c, line = 71

> Also do you remember any abnormal event (unclean shut-down, panic,
> power-lost, etc) *before* this issue pops out ?

Yes, I posted a few things about that recently. The cluster was dying in 
kernel panic until I updated all of them to be identical again. Since then, 
this node has been having these problems. I have also noticed that cman never 
shuts down correctly when I reboot nodes and that there is a lot of garbage 
(for lack of better word) about volume group information which no longer 
exists when I reboot nodes.

Last but not least, I wasn't sure what to post here so I decided I better post 
more than not enough.

compdev rc.sysinit: Checking root filesystem succeeded
compdev kernel: IP route cache hash table entries: 32768 (order: 5, 131072 
bytes)
compdev rc.sysinit: Remounting root filesystem in read-write mode:  succeeded
compdev kernel: TCP established hash table entries: 131072 (order: 8, 1048576 
bytes)
compdev lvm.static:
compdev kernel: TCP bind hash table entries: 131072 (order: 9, 3670016 bytes)
compdev lvm.static: connect() failed on local socket: Connection refused
compdev kernel: TCP: Hash tables configured (established 131072 bind 131072)
compdev lvm.static:   WARNING: Falling back to local file-based locking.
compdev kernel: Initializing IPsec netlink socket
compdev lvm.static:   Volume Groups with the clustered attribute will be 
inaccessible.
compdev kernel: NET: Registered protocol family 1
compdev lvm.static:   1 logical volume(s) in volume group VolGroup03 now 
active
compdev kernel: NET: Registered protocol family 17
compdev lvm.static:   1 logical volume(s) in volume group VolGroup02 now 
active
compdev kernel: Freeing unused kernel memory: 168k freed
compdev lvm.static:   1 logical volume(s) in volume group VolGroup01 now 
active
compdev kernel: SCSI subsystem initialized
compdev rc.sysinit: Setting up Logical Volume Management: succeeded
compdev kernel: QLogic Fibre Channel HBA Driver
compdev rc.sysinit: Checking filesystems succeeded
compdev kernel: qla2200 0000:00:11.0: Found an ISP2200, irq 11, iobase 
0xe0816000
compdev rc.sysinit: Mounting local filesystems:  succeeded
compdev kernel: qla2200 0000:00:11.0: Configuring PCI space...
compdev rc.sysinit: Enabling local filesystem quotas:  succeeded
compdev kernel: qla2200 0000:00:11.0: Configure NVRAM parameters...
compdev rc.sysinit: Enabling swap space:  succeeded
compdev kernel: qla2200 0000:00:11.0: Verifying loaded RISC code...
compdev init: Entering runlevel: 3
compdev kernel: qla2200 0000:00:11.0: LIP reset occured (0).
compdev microcode_ctl: microcode_ctl startup succeeded
compdev kernel: qla2200 0000:00:11.0: Waiting for LIP to complete...
compdev vgchange:   connect() failed on local socket: Connection refused
compdev kernel: qla2200 0000:00:11.0: LOOP UP detected (1 Gbps).
compdev vgchange:   connect() failed on local socket: Connection refused
compdev kernel: qla2200 0000:00:11.0: Topology - (F_Port), Host Loop address 
0xffff
compdev vgchange:
compdev kernel: scsi0 : qla2xxx
compdev vgchange:   WARNING: Falling back to local file-based locking.
compdev kernel: qla2200 0000:00:11.0:
compdev vgchange:   Volume Groups with the clustered attribute will be 
inaccessible.
compdev kernel:  QLogic Fibre Channel HBA Driver: 8.01.04-d8
compdev vgchange: Volume group "WARNING:" not found
compdev kernel:   QLogic QLA22xx -
compdev lvm2-monitor: Starting monitoring for VG WARNING:: failed
compdev kernel:   ISP2200: PCI (33 MHz) @ 0000:00:11.0 hdma-, host#=0, 
fw=2.02.08 TP
compdev vgchange:
compdev kernel:   Vendor: MYLEX     Model: DACARMRB          Rev: 7775
compdev vgchange: connect() failed on local socket: Connection refused
compdev kernel:   Type:   Direct-Access                      ANSI SCSI 
revision: 02
compdev vgchange:   connect() failed on local socket: Connection refused
compdev kernel: qla2200 0000:00:11.0: scsi(0:0:0:0): Enabled tagged queuing, 
queue depth 16.
compdev vgchange:
compdev kernel: SCSI device sda: 1013760000 512-byte hdwr sectors (519045 MB)
compdev vgchange:   WARNING: Falling back to local file-based locking.
compdev kernel: SCSI device sda: drive cache: write back
compdev vgchange:   Volume Groups with the clustered attribute will be 
inaccessible.
compdev kernel: SCSI device sda: 1013760000 512-byte hdwr sectors (519045 MB)
compdev vgchange: Volume group "Falling" not found
compdev kernel: SCSI device sda: drive cache: write back
compdev lvm2-monitor: Starting monitoring for VG Falling: failed
compdev kernel:  sda:
compdev vgchange:   connect() failed on local socket: Connection refused
compdev kernel: Attached scsi disk sda at scsi0, channel 0, id 0, lun 0
compdev vgchange:   connect() failed on local socket: Connection refused
compdev kernel:   Vendor: MYLEX     Model: DACARMRB          Rev: 7775
compdev vgchange:
compdev kernel:   Type:   Direct-Access                      ANSI SCSI 
revision: 02
compdev vgchange:   WARNING: Falling back to local file-based locking.
compdev kernel: qla2200 0000:00:11.0: scsi(0:0:0:1): Enabled tagged queuing, 
queue depth 16.
compdev vgchange:   Volume Groups with the clustered attribute will be 
inaccessible.
compdev kernel: SCSI device sdb: 1013760000 512-byte hdwr sectors (519045 MB)
compdev vgchange: Volume group "back" not found
compdev kernel: SCSI device sdb: drive cache: write back
compdev lvm2-monitor: Starting monitoring for VG back: failed
compdev kernel: SCSI device sdb: 1013760000 512-byte hdwr sectors (519045 MB)
compdev vgchange:   connect() failed on local socket: Connection refused
compdev kernel: SCSI device sdb: drive cache: write back
compdev vgchange:   connect() failed on local socket: Connection refused
compdev kernel:  sdb:
compdev vgchange:
compdev kernel: Attached scsi disk sdb at scsi0, channel 0, id 0, lun 1
compdev vgchange:   WARNING: Falling back to local file-based locking.
compdev kernel:   Vendor: MYLEX     Model: DACARMRB          Rev: 7775
compdev vgchange:   Volume Groups with the clustered attribute will be 
inaccessible.
compdev kernel:   Type:   Direct-Access                      ANSI SCSI 
revision: 02
compdev vgchange: Volume group "to" not found
compdev kernel: qla2200 0000:00:11.0: scsi(0:0:0:2): Enabled tagged queuing, 
queue depth 16.
compdev lvm2-monitor: Starting monitoring for VG to: failed
compdev kernel: SCSI device sdc: 997449728 512-byte hdwr sectors (510694 MB)
compdev vgchange:
compdev kernel: SCSI device sdc: drive cache: write back
compdev vgchange: connect() failed on local socket: Connection refused
compdev kernel: SCSI device sdc: 997449728 512-byte hdwr sectors (510694 MB)
compdev vgchange:   connect() failed on local socket: Connection refused
compdev kernel: SCSI device sdc: drive cache: write back
compdev vgchange:
compdev kernel:  sdc:
compdev vgchange:   WARNING: Falling back to local file-based locking.
compdev kernel: Attached scsi disk sdc at scsi0, channel 0, id 0, lun 2
compdev vgchange:   Volume Groups with the clustered attribute will be 
inaccessible.
compdev vgchange: Volume group "local" not found
compdev kernel: device-mapper: 4.5.5-ioctl (2006-12-01) initialised: 
dm-devel redhat com
compdev lvm2-monitor: Starting monitoring for VG local: failed
compdev kernel: kjournald starting.  Commit interval 5 seconds
compdev vgchange:   connect() failed on local socket: Connection refused
compdev kernel: EXT3-fs: mounted filesystem with ordered data mode.
compdev vgchange:   connect() failed on local socket: Connection refused
compdev kernel: SELinux:  Disabled at runtime.
compdev vgchange:
compdev kernel: SELinux:  Unregistering netfilter hooks
compdev vgchange:   WARNING: Falling back to local file-based locking.
compdev kernel: inserting floppy driver for 2.6.9-55.0.12.EL.XOS.1
compdev vgchange:   Volume Groups with the clustered attribute will be 
inaccessible.
compdev kernel: Floppy drive(s): fd0 is 1.44M
compdev vgchange: Volume group "file-based" not found
compdev kernel: FDC 0 is a post-1991 82077
compdev lvm2-monitor: Starting monitoring for VG file-based: failed
compdev kernel: e100: Intel(R) PRO/100 Network Driver, 3.5.10-k2-NAPI
compdev vgchange:   connect() failed on local socket: Connection refused
compdev kernel: e100: Copyright(c) 1999-2005 Intel Corporation
compdev vgchange:   connect() failed on local socket: Connection refused
compdev vgchange:
compdev kernel: e100: eth0: e100_probe: addr 0xfebfe000, irq 5, MAC addr 
00:20:94:10:43:67
compdev vgchange:   WARNING: Falling back to local file-based locking.
compdev vgchange:   Volume Groups with the clustered attribute will be 
inaccessible.
compdev kernel: e100: eth1: e100_probe: addr 0xfebfd000, irq 11, MAC addr 
00:20:94:10:43:68
compdev vgchange: Volume group "locking." not found
compdev kernel: USB Universal Host Controller Interface driver v2.2
compdev lvm2-monitor: Starting monitoring for VG locking.: failed
compdev kernel: PCI: Enabling device 0000:00:07.2 (0000 -> 0001)
compdev vgchange:   connect() failed on local socket: Connection refused
compdev kernel: PCI: No IRQ known for interrupt pin D of device 0000:00:07.2. 
Please try using pci=biosi
rq.
compdev vgchange:   connect() failed on local socket: Connection refused
compdev kernel: uhci_hcd 0000:00:07.2: Found HC with no IRQ.  Check BIOS/PCI 
0000:00:07.2 setup!
compdev vgchange:
compdev kernel: md: Autodetecting RAID arrays.
compdev vgchange:   WARNING: Falling back to local file-based locking.
compdev kernel: md: autorun ...
compdev vgchange:   Volume Groups with the clustered attribute will be 
inaccessible.
compdev kernel: md: ... autorun DONE.
compdev vgchange: Volume group "Volume" not found
compdev kernel: EXT3 FS on hda1, internal journal
compdev lvm2-monitor: Starting monitoring for VG Volume: failed
compdev kernel: Adding 787176k swap on /dev/hda2.  Priority:-1 extents:1
compdev vgchange:
compdev kernel: IA-32 Microcode Update Driver: v1.14 <tigran veritas com>
compdev vgchange: connect() failed on local socket: Connection refused
compdev kernel: microcode: CPU0 updated from revision 0x7 to 0x8, date = 
05052000
compdev vgchange:   connect() failed on local socket: Connection refused
compdev kernel: IA-32 Microcode Update Driver v1.14 unregistered
compdev vgchange:
compdev kernel: ip_tables: (C) 2000-2002 Netfilter core team
compdev vgchange:   WARNING: Falling back to local file-based locking.
compdev kernel: ip_tables: (C) 2000-2002 Netfilter core team
compdev vgchange:   Volume Groups with the clustered attribute will be 
inaccessible.
compdev kernel: e100: eth0: e100_watchdog: link up, 100Mbps, full-duplex
compdev vgchange: Volume group "Groups" not found
compdev kernel: NET: Registered protocol family 10
compdev lvm2-monitor: Starting monitoring for VG Groups: failed
compdev kernel: Disabled Privacy Extensions on device c0386e60(lo)
compdev vgchange:   connect() failed on local socket: Connection refused
compdev kernel: IPv6 over IPv4 tunneling driver
compdev vgchange:   connect() failed on local socket: Connection refused
compdev vgchange:
compdev kernel: CMAN 2.6.9-50.2.0.6.XOS.1 (built Nov 15 2007 12:03:01) 
installed
compdev vgchange:   WARNING: Falling back to local file-based locking.
compdev kernel: NET: Registered protocol family 30
compdev vgchange:   Volume Groups with the clustered attribute will be 
inaccessible.
compdev kernel: DLM 2.6.9-46.16.0.12.XOS.1 (built Nov 15 2007 12:27:30) 
installed
compdev vgchange: Volume group "with" not found
compdev lvm2-monitor: Starting monitoring for VG with: failed
compdev vgchange:   connect() failed on local socket: Connection refused
compdev vgchange:   connect() failed on local socket: Connection refused
compdev vgchange:
compdev vgchange:   WARNING: Falling back to local file-based locking.
compdev vgchange:   Volume Groups with the clustered attribute will be 
inaccessible.
compdev vgchange: Volume group "the" not found
compdev lvm2-monitor: Starting monitoring for VG the: failed
compdev vgchange:
compdev vgchange: connect() failed on local socket: Connection refused
compdev vgchange:   connect() failed on local socket: Connection refused
compdev vgchange:
compdev vgchange:   WARNING: Falling back to local file-based locking.
compdev vgchange:   Volume Groups with the clustered attribute will be 
inaccessible.
compdev vgchange: Volume group "clustered" not found
compdev lvm2-monitor: Starting monitoring for VG clustered: failed
compdev vgchange:   connect() failed on local socket: Connection refused
compdev vgchange:   connect() failed on local socket: Connection refused
compdev vgchange:
compdev vgchange:   WARNING: Falling back to local file-based locking.
compdev vgchange:   Volume Groups with the clustered attribute will be 
inaccessible.
compdev vgchange: Volume group "attribute" not found
compdev lvm2-monitor: Starting monitoring for VG attribute: failed
compdev vgchange:   connect() failed on local socket: Connection refused
compdev vgchange:   connect() failed on local socket: Connection refused
compdev vgchange:
compdev vgchange:   WARNING: Falling back to local file-based locking.
compdev vgchange:   Volume Groups with the clustered attribute will be 
inaccessible.
compdev vgchange: Volume group "will" not found
compdev lvm2-monitor: Starting monitoring for VG will: failed
compdev vgchange:
compdev vgchange: connect() failed on local socket: Connection refused
compdev vgchange:   connect() failed on local socket: Connection refused
compdev vgchange:
compdev vgchange:   WARNING: Falling back to local file-based locking.
compdev vgchange:   Volume Groups with the clustered attribute will be 
inaccessible.
compdev vgchange: Volume group "be" not found
compdev lvm2-monitor: Starting monitoring for VG be: failed
compdev vgchange:   connect() failed on local socket: Connection refused
compdev vgchange:   connect() failed on local socket: Connection refused
compdev vgchange:
compdev vgchange:   WARNING: Falling back to local file-based locking.
compdev vgchange:   Volume Groups with the clustered attribute will be 
inaccessible.
compdev vgchange: Volume group "inaccessible." not found
compdev lvm2-monitor: Starting monitoring for VG inaccessible.: failed
compdev vgchange:   connect() failed on local socket: Connection refused
compdev vgchange:   connect() failed on local socket: Connection refused
compdev vgchange:   WARNING: Falling back to local file-based locking.
compdev vgchange:   Volume Groups with the clustered attribute will be 
inaccessible.
compdev vgchange:   1 logical volume(s) in volume group "VolGroup01" monitored
compdev lvm2-monitor: Starting monitoring for VG VolGroup01: succeeded
compdev vgchange:
compdev vgchange: connect() failed on local socket: Connection refused
compdev vgchange:   connect() failed on local socket: Connection refused
compdev vgchange:   WARNING: Falling back to local file-based locking.
compdev vgchange:   Volume Groups with the clustered attribute will be 
inaccessible.
compdev vgchange:   1 logical volume(s) in volume group "VolGroup02" monitored
compdev lvm2-monitor: Starting monitoring for VG VolGroup02: succeeded
compdev vgchange:
compdev vgchange: connect() failed on local socket: Connection refused
compdev vgchange:   connect() failed on local socket: Connection refused
compdev vgchange:   WARNING: Falling back to local file-based locking.
compdev vgchange:   Volume Groups with the clustered attribute will be 
inaccessible.
compdev vgchange:   1 logical volume(s) in volume group "VolGroup03" monitored
compdev lvm2-monitor: Starting monitoring for VG VolGroup03: succeeded
compdev kudzu:  succeeded
compdev sysctl: net.ipv4.ip_forward = 0
compdev sysctl: net.ipv4.conf.default.rp_filter = 1
compdev sysctl: net.ipv4.conf.default.accept_source_route = 0
compdev sysctl: kernel.sysrq = 0
compdev sysctl: kernel.core_uses_pid = 1
compdev sysctl: kernel.panic_on_oops = 1
compdev network: Setting network parameters:  succeeded
compdev network: Bringing up loopback interface:  succeeded
compdev network: Bringing up interface eth0:  succeeded
compdev ccsd[2458]: Remote copy of cluster.conf is from quorate node.
compdev ccsd[2458]:  Local version # : 80
compdev ccsd[2458]:  Remote version #: 80
compdev kernel: CMAN: Waiting to join or form a Linux-cluster
compdev kernel: CMAN: sending membership request
compdev ccsd[2458]: Connected to cluster infrastruture via: CMAN/SM Plugin 
v1.1.7.4
compdev ccsd[2458]: Initial status:: Inquorate
compdev kernel: CMAN: got node cweb93
compdev kernel: CMAN: got node cweb94
compdev kernel: CMAN: got node cweb92
compdev kernel: CMAN: got node img62
compdev ccsd[2458]: Cluster is quorate.  Allowing connections.
compdev kernel: CMAN: quorum regained, resuming activity
compdev cman: startup succeeded
compdev fenced: startup succeeded
compdev clvmd: Cluster LVM daemon started - connected to CMAN
compdev clvmd: clvmd startup succeeded
compdev vgchange:   1 logical volume(s) in volume group "VolGroup03" now 
active
compdev vgchange:   1 logical volume(s) in volume group "VolGroup02" now 
active
compdev vgchange:   1 logical volume(s) in volume group "VolGroup01" now 
active
compdev clvmd: Activating VGs: succeeded
compdev netfs: Mounting other filesystems:  succeeded
compdev kernel: Lock_Harness 2.6.9-72.2.0.9.XOS.1 (built Nov 15 2007 12:30:46) 
installed
compdev kernel: GFS 2.6.9-72.2.0.9.XOS.1 (built Nov 15 2007 12:31:07) 
installed
compdev kernel: GFS: Trying to join cluster "lock_dlm", "vgcomp:qm"
compdev kernel: Lock_DLM (built Nov 15 2007 12:30:48) installed
compdev kernel: GFS: fsid=vgcomp:qm.1: Joined cluster. Now mounting FS...
compdev kernel: GFS: fsid=vgcomp:qm.1: jid=1: Trying to acquire journal 
lock...
compdev kernel: GFS: fsid=vgcomp:qm.1: jid=1: Looking at journal...
compdev kernel: GFS: fsid=vgcomp:qm.1: jid=1: Done
compdev kernel: GFS: Trying to join cluster "lock_dlm", "vgcomp:web"
compdev kernel: GFS: fsid=vgcomp:web.3: Joined cluster. Now mounting FS...
compdev kernel: GFS: fsid=vgcomp:web.3: jid=3: Trying to acquire journal 
lock...
compdev kernel: GFS: fsid=vgcomp:web.3: jid=3: Looking at journal...
compdev kernel: GFS: fsid=vgcomp:web.3: jid=3: Done
compdev kernel: GFS: fsid=vgcomp:web.3: Scanning for log elements...
compdev kernel: GFS: fsid=vgcomp:web.3: Found 1 unlinked inodes
compdev kernel: GFS: fsid=vgcomp:web.3: Found quota changes for 0 IDs
compdev kernel: GFS: fsid=vgcomp:web.3: Done
compdev gfs: Mounting GFS filesystems:  succeeded




[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]