[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

[Linux-cluster] accessing files on gfs caused fatal error, gfs on both boxes crumbles..



The gfs share was responding fine for the most part. Not sure if this is related to the postfix question I asked, but a I was moving files back and forth manually (just getting the boxes setup running the same things and configured the same way), I placed something on the share from server 2 (goumang), then when i tried to access it via box 1(rushou) things went bad.

I had placed the source for one of the apps that I am going to install (rather small thing) on the share, ran ./configure and voila..

[root rushou rabl_client-1.0.0]# ./configure \
> --prefix=/ \
> --exec-prefix=/usr \
> --includedir=/usr/include
-bash: ./configure: Permission denied
[root rushou rabl_client-1.0.0]#
[root rushou rabl_client-1.0.0]# ls -l
ls: .: Input/output error
[root rushou rabl_client-1.0.0]# ls -lZ
ls: .: Input/output error

I tried restarting gfs2 on rushou, and it worked once, went back into the directory, ran it again (hey, gotta replicate it right?) and same thing, except on restart i got this:
[root rushou mnt]# service gfs2 restart
Unmounting GFS2 filesystems: [ OK ]
FATAL: Module lock_dlm is in use.
FATAL: Module gfs2 is in use.
Mounting GFS2 filesystems: /sbin/mount.gfs2: error: jid is -2, try rw
/sbin/mount.gfs2: error mounting lockproto lock_dlm
[FAILED]

Then when i tried to do anything on the gfs mount on goumang... LOCK.

So maybe running a configure from the gfs drive wasnt so bright... but in theory with the load i plan on hitting this thing with (mail server) I would hope a small configure setup wouldnt cause to much issue.

i've attached the info from /var/log/messages

[root rushou /]# uname -a
Linux rushou.sgc 2.6.18-1.2849.fc6 #1 SMP Fri Nov 10 12:36:14 EST 2006 i686 i686 i386 GNU/Linux
[root rushou /]# rpm -qa | grep gfs
gfs2-utils-0.1.7-1.fc6
[root rushou /]# rpm -qa | grep cman
cman-2.0.18-2.fc6
[root rushou /]# rpm -qa | grep lvm
lvm2-2.02.06-4
lvm2-cluster-2.02.06-1.5
system-config-lvm-1.0.18-1.2.FC6


any other information that would be useful?

-greg

--
http://www.gvtc.com
--
“While it is possible to change without improving, it is impossible to improve without changing.” -anonymous

“only he who attempts the absurd can achieve the impossible.” -anonymous

Nov 28 23:32:14 rushou kernel: GFS2: fsid=outMail:data.0: fatal: filesystem consistency error
Nov 28 23:32:14 rushou kernel: GFS2: fsid=outMail:data.0:   inode = 1086093 3284792
Nov 28 23:32:14 rushou kernel: GFS2: fsid=outMail:data.0:   function = gfs2_drevalidate, file = fs/gfs2/ops_dentry.c, line = 80
Nov 28 23:32:14 rushou kernel: GFS2: fsid=outMail:data.0: about to withdraw from the cluster
Nov 28 23:32:14 rushou kernel: GFS2: fsid=outMail:data.0: waiting for outstanding I/O
Nov 28 23:32:14 rushou kernel: GFS2: fsid=outMail:data.0: telling LM to withdraw
Nov 28 23:32:14 rushou kernel: GFS2: fsid=outMail:data.0: withdrawn
Nov 28 23:32:14 rushou kernel:  [<c0405876>] dump_trace+0x69/0x1af
Nov 28 23:32:14 rushou kernel:  [<c04059d4>] show_trace_log_lvl+0x18/0x2c
Nov 28 23:32:14 rushou kernel:  [<c0405f83>] show_trace+0xf/0x11
Nov 28 23:32:14 rushou kernel:  [<c0406080>] dump_stack+0x15/0x17
Nov 28 23:32:14 rushou kernel:  [<f8e88063>] gfs2_lm_withdraw+0x9a/0xa5 [gfs2]
Nov 28 23:32:14 rushou kernel:  [<f8e9a380>] gfs2_consist_inode_i+0x6f/0x75 [gfs2]
Nov 28 23:32:14 rushou kernel:  [<f8e8de87>] gfs2_drevalidate+0x13e/0x1ef [gfs2]
Nov 28 23:32:14 rushou kernel:  [<c0476e90>] do_lookup+0x114/0x15a
Nov 28 23:32:14 rushou kernel:  [<c0478c37>] __link_path_walk+0x8ad/0xd73
Nov 28 23:32:14 rushou kernel:  [<c0479146>] link_path_walk+0x49/0xbd
Nov 28 23:32:14 rushou kernel:  [<c0479523>] do_path_lookup+0x21a/0x26b
Nov 28 23:32:14 rushou kernel:  [<c0479ce8>] __user_walk_fd+0x2f/0x40
Nov 28 23:32:14 rushou kernel:  [<c0473803>] vfs_stat_fd+0x19/0x40
Nov 28 23:32:14 rushou kernel:  [<c04738b7>] sys_stat64+0xf/0x23
Nov 28 23:32:14 rushou kernel:  [<c0404027>] syscall_call+0x7/0xb
Nov 28 23:32:14 rushou kernel: DWARF2 unwinder stuck at syscall_call+0x7/0xb
Nov 28 23:32:14 rushou kernel: Leftover inexact backtrace:
Nov 28 23:32:14 rushou kernel:  =======================
Nov 28 23:33:29 rushou multipathd: dm-1: umount map (uevent)
Nov 28 23:33:34 rushou kernel: GFS2: fsid=: Trying to join cluster "lock_dlm", "outMail:data"
Nov 28 23:33:34 rushou kernel: dlm: data: recover 1
Nov 28 23:33:34 rushou kernel: GFS2: fsid=outMail:data.0: Joined cluster. Now mounting FS...
Nov 28 23:33:34 rushou kernel: dlm: data: add member 1
Nov 28 23:33:34 rushou kernel: dlm: data: add member 2
Nov 28 23:33:34 rushou kernel: dlm: data: total members 2 error 0
Nov 28 23:33:34 rushou kernel: dlm: data: dlm_recover_directory
Nov 28 23:33:34 rushou kernel: dlm: data: dlm_recover_directory 18762 entries
Nov 28 23:33:34 rushou kernel: dlm: data: recover 1 done: 97 ms
Nov 28 23:33:34 rushou kernel: GFS2: fsid=outMail:data.0: jid=0, already locked for use
Nov 28 23:33:34 rushou kernel: GFS2: fsid=outMail:data.0: jid=0: Looking at journal...
Nov 28 23:33:34 rushou kernel: GFS2: fsid=outMail:data.0: jid=0: Done
Nov 28 23:33:54 rushou kernel: GFS2: fsid=outMail:data.0: fatal: filesystem consistency error
Nov 28 23:33:54 rushou kernel: GFS2: fsid=outMail:data.0:   inode = 1086093 3284792
Nov 28 23:33:54 rushou kernel: GFS2: fsid=outMail:data.0:   function = gfs2_drevalidate, file = fs/gfs2/ops_dentry.c, line = 80
Nov 28 23:33:54 rushou kernel: GFS2: fsid=outMail:data.0: about to withdraw from the cluster
Nov 28 23:33:54 rushou kernel: GFS2: fsid=outMail:data.0: waiting for outstanding I/O
Nov 28 23:33:54 rushou kernel: GFS2: fsid=outMail:data.0: telling LM to withdraw
Nov 28 23:33:54 rushou kernel: GFS2: fsid=outMail:data.0: withdrawn
Nov 28 23:33:54 rushou kernel:  [<c0405876>] dump_trace+0x69/0x1af
Nov 28 23:33:54 rushou kernel:  [<c04059d4>] show_trace_log_lvl+0x18/0x2c
Nov 28 23:33:54 rushou kernel:  [<c0405f83>] show_trace+0xf/0x11
Nov 28 23:33:54 rushou kernel:  [<c0406080>] dump_stack+0x15/0x17
Nov 28 23:33:54 rushou kernel:  [<f8e88063>] gfs2_lm_withdraw+0x9a/0xa5 [gfs2]
Nov 28 23:33:54 rushou kernel:  [<f8e9a380>] gfs2_consist_inode_i+0x6f/0x75 [gfs2]
Nov 28 23:33:54 rushou kernel:  [<f8e8de87>] gfs2_drevalidate+0x13e/0x1ef [gfs2]
Nov 28 23:33:54 rushou kernel:  [<c0476e90>] do_lookup+0x114/0x15a
Nov 28 23:33:54 rushou kernel:  [<c0478c37>] __link_path_walk+0x8ad/0xd73
Nov 28 23:33:54 rushou kernel:  [<c0479146>] link_path_walk+0x49/0xbd
Nov 28 23:33:54 rushou kernel:  [<c0479523>] do_path_lookup+0x21a/0x26b
Nov 28 23:33:54 rushou kernel:  [<c0479ce8>] __user_walk_fd+0x2f/0x40
Nov 28 23:33:54 rushou kernel:  [<c0473803>] vfs_stat_fd+0x19/0x40
Nov 28 23:33:54 rushou kernel:  [<c04738b7>] sys_stat64+0xf/0x23
Nov 28 23:33:54 rushou kernel:  [<c0404027>] syscall_call+0x7/0xb
Nov 28 23:33:54 rushou kernel: DWARF2 unwinder stuck at syscall_call+0x7/0xb
Nov 28 23:33:54 rushou kernel: Leftover inexact backtrace:
Nov 28 23:33:54 rushou kernel:  =======================
Nov 28 23:34:18 rushou multipathd: dm-1: umount map (uevent)
Nov 28 23:32:14 goumang kernel: GFS2: fsid=outMail:data.1: jid=0: Trying to acquire journal lock...
Nov 28 23:32:14 goumang kernel: dlm: data: recover 3
Nov 28 23:32:14 goumang kernel: dlm: data: remove member 2
Nov 28 23:32:14 goumang kernel: dlm: data: total members 1 error 0
Nov 28 23:32:14 goumang kernel: dlm: data: dlm_recover_directory
Nov 28 23:32:14 goumang kernel: dlm: data: dlm_recover_directory 37412 entries
Nov 28 23:32:14 goumang kernel: dlm: data: pre recover waiter lkid 1480322 type 11 flags 0
Nov 28 23:32:14 goumang kernel: dlm: data: dlm_purge_locks
Nov 28 23:32:14 goumang kernel: dlm: data: dlm_recover_masters
Nov 28 23:32:14 goumang kernel: dlm: data: dlm_recover_masters 44 resources
Nov 28 23:32:14 goumang kernel: dlm: data: dlm_recover_locks
Nov 28 23:32:14 goumang kernel: dlm: data: dlm_recover_locks 0 locks
Nov 28 23:32:14 goumang kernel: dlm: data: dlm_recover_rsbs
Nov 28 23:32:14 goumang kernel: dlm: data: dlm_recover_rsbs 37456 rsbs
Nov 28 23:32:14 goumang kernel: dlm: data: recover_waiters_post 1480322 type 11 flags 0        9               0
Nov 28 23:32:14 goumang kernel: GFS2: fsid=outMail:data.1: jid=0: Looking at journal...
Nov 28 23:32:14 goumang kernel: dlm: data: recover 3 done: 186 ms
Nov 28 23:32:14 goumang kernel: GFS2: fsid=outMail:data.1: jid=0: Done
Nov 28 23:33:34 goumang kernel: dlm: data: recover 5
Nov 28 23:33:34 goumang kernel: dlm: data: add member 2
Nov 28 23:33:34 goumang kernel: dlm: data: total members 2 error 0
Nov 28 23:33:34 goumang kernel: dlm: data: dlm_recover_directory
Nov 28 23:33:34 goumang kernel: dlm: data: dlm_recover_directory 18638 entries
Nov 28 23:33:34 goumang kernel: dlm: data: recover 5 done: 105 ms
Nov 28 23:33:54 goumang kernel: GFS2: fsid=outMail:data.1: jid=0: Trying to acquire journal lock...
Nov 28 23:33:54 goumang kernel: GFS2: fsid=outMail:data.1: jid=0: Busy
Nov 28 23:33:54 goumang kernel: dlm: data: recover 7
Nov 28 23:33:54 goumang kernel: dlm: data: remove member 2
Nov 28 23:33:54 goumang kernel: dlm: data: total members 1 error 0
Nov 28 23:33:54 goumang kernel: dlm: data: dlm_recover_directory
Nov 28 23:33:54 goumang kernel: dlm: data: dlm_recover_directory 37400 entries
Nov 28 23:33:54 goumang kernel: dlm: data: dlm_purge_locks
Nov 28 23:33:54 goumang kernel: dlm: data: dlm_recover_masters
Nov 28 23:33:54 goumang kernel: dlm: data: dlm_recover_masters 0 resources
Nov 28 23:33:54 goumang kernel: dlm: data: dlm_recover_locks
Nov 28 23:33:54 goumang kernel: dlm: data: dlm_recover_locks 0 locks
Nov 28 23:33:54 goumang kernel: dlm: data: dlm_recover_rsbs
Nov 28 23:33:54 goumang kernel: dlm: data: dlm_recover_rsbs 37400 rsbs
Nov 28 23:33:54 goumang kernel: dlm: data: recover 7 done: 103 ms

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]