[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [Linux-cluster] Cluster fail over fails when umounting fs



hi,
     	plz check the following
	1) e2fsck with -c /dev/sda1
	2) hard disk with vendor supply hardware/SAN health monitoring utility
	3) Is it any network flood like DOS attack in your network.

regards
karthikeyan.N

Neil Watson wrote:
I'm build a cluster that runs a DB2 service.  The cluster has 2 nodes
in an active standby configuration.  I am now performing fail over
tests.

Shared resources:
DB2 controlled by /etc/init.d/db2 start stop script.
Floating IP address.
/db2 ext3 file system located on a SAN and connected via HBA.

Nodes are fenced with ILO cards.

Nodes are running AS4 x86_64 with the Redhat Cluster Suite.  RPMs are up
to date.

Procedure:

1. Connect to DB2 remotely and begin a long SQL insert program.
2. While the inserts a being performed, disconnected the fibre cable
from the HBA, on the active node.
3. Examine the system logs an observe for fail over.

Observations:

1. Cluster does not fail over to standby node.  Service becomes
unavailable.
2. The log files of the active node report a 'generic error' about the status
of the shared file system.

Aug 16 15:32:37 caesar kernel: qla2300 0000:06:01.0: LOOP DOWN detected (2).
Aug 16 15:32:45 caesar kernel: SCSI error : <0 0 0 1> return code = 0x10000
Aug 16 15:32:45 caesar kernel: end_request: I/O error, dev sda, sector 15839
Aug 16 15:32:45 caesar kernel: SCSI error : <0 0 0 1> return code = 0x10000
Aug 16 15:32:45 caesar kernel: end_request: I/O error, dev sda, sector 15847
Aug 16 15:32:45 caesar kernel: SCSI error : <0 0 0 1> return code = 0x10000
Aug 16 15:32:45 caesar kernel: end_request: I/O error, dev sda, sector 15855 Aug 16 15:32:45 caesar kernel: Buffer I/O error on device sda1, logical block 1974
Aug 16 15:32:45 caesar kernel: lost page write due to I/O error on sda1
Aug 16 15:32:45 caesar kernel: SCSI error : <0 0 0 1> return code = 0x10000
Aug 16 15:32:45 caesar kernel: end_request: I/O error, dev sda, sector 103813199 Aug 16 15:32:45 caesar kernel: Buffer I/O error on device sda1, logical block 12976642
Aug 16 15:32:45 caesar kernel: lost page write due to I/O error on sda1
Aug 16 15:32:45 caesar kernel: Aborting journal on device sda1.
Aug 16 15:32:45 caesar kernel: ext3_abort called.
Aug 16 15:32:45 caesar kernel: EXT3-fs error (device sda1): ext3_journal_start_sb: Detected aborted journal
Aug 16 15:32:45 caesar kernel: Remounting filesystem read-only
Aug 16 15:32:47 caesar kernel: SCSI error : <0 0 0 1> return code = 0x10000
Aug 16 15:32:47 caesar kernel: end_request: I/O error, dev sda, sector 8279
Aug 16 15:32:47 caesar kernel: Buffer I/O error on device sda1, logical block 1027
Aug 16 15:32:47 caesar kernel: lost page write due to I/O error on sda1
Aug 16 15:32:47 caesar kernel: SCSI error : <0 0 0 1> return code = 0x10000
Aug 16 15:32:47 caesar kernel: end_request: I/O error, dev sda, sector 103546959 Aug 16 15:32:47 caesar kernel: Buffer I/O error on device sda1, logical block 12943362
Aug 16 15:32:47 caesar kernel: lost page write due to I/O error on sda1
Aug 16 15:32:48 caesar clurgmgrd[5159]: <notice> status on fs "db2" returned 1 (generic error)
Aug 16 15:32:48 caesar clurgmgrd[5159]: <notice> Stopping service db2
Aug 16 15:32:48 caesar clurgmgrd: [5159]: <info> Executing /etc/rc.d/init.d/db2 stop Aug 16 15:32:48 caesar su(pam_unix)[1227]: session opened for user dwapinst by (uid=0)
Aug 16 15:32:49 caesar su:
Aug 16 15:32:49 caesar su: Instance  : dwapinst
Aug 16 15:32:49 caesar su: DB2 State : Available
Aug 16 15:32:49 caesar su(pam_unix)[1227]: session closed for user dwapinst
Aug 16 15:32:49 caesar db2:  succeeded
Aug 16 15:32:49 caesar su(pam_unix)[1322]: session opened for user dwapinst by (uid=0) Aug 16 15:36:51 caesar su(pam_unix)[5473]: session opened for user root by nhwatson(uid=0) Aug 16 15:36:55 caesar su(pam_unix)[6000]: session opened for user dwapinst by nhwatson(uid=0)
Aug 16 15:36:55 caesar su:
Aug 16 15:36:55 caesar su: Instance  : dwapinst
Aug 16 15:36:55 caesar su: DB2 State : Operable
Aug 16 15:36:55 caesar su(pam_unix)[6000]: session closed for user dwapinst
Aug 16 15:36:55 caesar db2:  failed
3. The are no log entries for this event on the standby node.

Why does the cluster fail during this test?  What does the 'generic error'
mean?



[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]