[Linux-cluster] uncontrolled entry
Vladimir Melnik
v.melnik at uplink.ua
Thu Oct 24 15:11:30 UTC 2013
Hi!
Thank you. That's how clister.conf looks like:
--- 8< ---
<?xml version="1.0"?>
<cluster name="ckvm1_pod1" config_version="5">
<clusternodes>
<clusternode name="bond0-101.host1.ckvm1.pod1.***.net" votes="1"
nodeid="1"><fence><method name="single"><device
name="host1_ipmi"/></method></fence></clusternode><clusternode
name="bond0-101.host2.ckvm1.pod1.***.net" votes="1"
nodeid="2"><fence><method name="single"><device
name="host2_ipmi"/></method></fence></clusternode><clusternode
name="bond0-101.host3.ckvm1.pod1.***.net" votes="1"
nodeid="3"><fence><method name="single"><device
name="host3_ipmi"/></method></fence></clusternode><clusternode
name="bond0-101.host4.ckvm1.pod1.***.net" votes="1"
nodeid="4"><fence><method name="single"><device
name="host4_ipmi"/></method></fence></clusternode></clusternodes>
<fencedevices>
<fencedevice name="host1_ipmi" agent="fence_ipmilan"
ipaddr="mc.host1.ckvm1.pod1.***.net" login="***" passwd="***"/><fencedevice
name="host2_ipmi" agent="fence_ipmilan" ipaddr="mc.host2.ckvm1.pod1.***.net"
login="***" passwd="***"/>="***"/><fencedevice name="host3_ipmi"
agent="fence_ipmilan" ipaddr="mc.host3.ckvm1.pod1.***.net" login="***"
passwd="***"/>="***"/><fencedevice name="host4_ipmi" agent="fence_ipmilan"
ipaddr="mc.host4.ckvm1.pod1.***.net" login="***"
passwd="***"/></fencedevices>
<rm>
<failoverdomains/>
<resources/>
</rm>
</cluster>
--- >8 ---
That's what I've got in logg on the affected node:
--- 8< ---
Oct 24 13:01:57 host1 kernel: sd 10:0:0:0: [sdb] Synchronizing SCSI cache
Oct 24 13:01:57 host1 kernel: connection6:0: detected conn error (1020)
Oct 24 13:01:58 host1 kernel: scsi11 : iSCSI Initiator over TCP/IP
Oct 24 13:01:58 host1 kernel: scsi 11:0:0:0: Direct-Access OSNEXUS
QUANTASTOR 300 PQ: 0 ANSI: 6
Oct 24 13:01:58 host1 kernel: sd 11:0:0:0: [sdc] 3478923509 512-byte logical
blocks: (1.78 TB/1.61 TiB)
Oct 24 13:01:58 host1 kernel: sd 11:0:0:0: [sdc] 4096-byte physical blocks
Oct 24 13:01:58 host1 kernel: sd 11:0:0:0: [sdc] Write Protect is off
Oct 24 13:01:58 host1 kernel: sd 11:0:0:0: Attached scsi generic sg2 type 0
Oct 24 13:01:58 host1 kernel: sd 11:0:0:0: [sdc] Write cache: enabled, read
cache: enabled, supports DPO and FUA
Oct 24 13:01:58 host1 kernel: sdc: unknown partition table
Oct 24 13:01:58 host1 kernel: sd 11:0:0:0: [sdc] Attached SCSI disk
Oct 24 13:01:58 host1 iscsid: Connection6:0 to [target:
iqn.2009-10.com.osnexus:4eb54cf4-3f64da4ba5413ecc:PSV4, portal:
192.168.***.***,3260] through [iface: default] is shutdown.
Oct 24 13:01:58 host1 iscsid: Connection7:0 to [target:
iqn.2009-10.com.osnexus:4eb54cf4-3f64da4ba5413ecc:PSV4, portal:
192.168.***.***,3260] through [iface: default] is operational now
Oct 24 13:02:01 host1 kernel: ------------[ cut here ]------------
Oct 24 13:02:01 host1 kernel: WARNING: at fs/buffer.c:1199
__brelse+0x3a/0x40() (Not tainted)
Oct 24 13:02:01 host1 kernel: Hardware name: System x3630 M3 -[7377W03]-
Oct 24 13:02:01 host1 kernel: VFS: brelse: Trying to free free buffer
Oct 24 13:02:01 host1 kernel: Modules linked in: gfs2 dlm configfs ext3 jbd
nfs fscache ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE
xt_state nfsd lockd nfs_acl auth_rpcgss
exportfs autofs4 sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf
bridge bonding 8021q garp stp llc xt_CHECKSUM iptable_mangle ipt_REJECT
iptable_filter iptable_nat nf_nat nf_conntrack_i
pv4 nf_conntrack nf_defrag_ipv4 ip_tables be2iscsi iscsi_boot_sysfs bnx2i
cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio ib_iser rdma_cm ib_cm iw_cm
ib_sa ib_mad ib_core ib_addr ipv6 iscsi_
tcp libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan tun
kvm_intel kvm ses enclosure sg e1000e igb cdc_ether usbnet mii microcode
serio_raw i2c_i801 i2c_core iTCO_wdt iTCO
_vendor_support ioatdma dca i7core_edac edac_core shpchp ext4 mbcache jbd2
sd_mod crc_t10dif pata_acpi ata_generic ata_piix megaraid_sas dm_mirror
dm_region_hash dm_log dm_mod [last unloaded:
configfs]
Oct 24 13:02:01 host1 kernel: Pid: 28581, comm: qemu-kvm Not tainted
2.6.32-279.5.1.el6.x86_64 #1
Oct 24 13:02:01 host1 kernel: Call Trace:
Oct 24 13:02:01 host1 kernel: [<ffffffff8106b747>] ?
warn_slowpath_common+0x87/0xc0
Oct 24 13:02:01 host1 kernel: [<ffffffff8106b836>] ?
warn_slowpath_fmt+0x46/0x50
Oct 24 13:02:01 host1 kernel: [<ffffffff8112a965>] ?
pagevec_lookup_tag+0x25/0x40
Oct 24 13:02:01 host1 kernel: [<ffffffff811ad48a>] ? __brelse+0x3a/0x40
Oct 24 13:02:01 host1 kernel: [<ffffffffa078bbc1>] ?
gfs2_block_map+0x1e1/0xf30 [gfs2]
Oct 24 13:02:01 host1 kernel: [<ffffffff81048fb5>] ?
gup_pte_range+0xe5/0x130
Oct 24 13:02:01 host1 kernel: [<ffffffff81049148>] ?
gup_pud_range+0x148/0x1d0
Oct 24 13:02:01 host1 kernel: [<ffffffffa07a1770>] ?
gfs2_get_block_direct+0x10/0x20 [gfs2]
Oct 24 13:02:01 host1 kernel: [<ffffffff811b689a>] ?
__blockdev_direct_IO_newtrunc+0x85a/0xb90
Oct 24 13:02:01 host1 kernel: [<ffffffff8112a965>] ?
pagevec_lookup_tag+0x25/0x40
Oct 24 13:02:01 host1 kernel: [<ffffffffa0799ca8>] ? do_promote+0x208/0x330
[gfs2]
Oct 24 13:02:01 host1 kernel: [<ffffffff811b6c2e>] ?
__blockdev_direct_IO+0x5e/0xd0
Oct 24 13:02:01 host1 kernel: [<ffffffffa07a1760>] ?
gfs2_get_block_direct+0x0/0x20 [gfs2]
Oct 24 13:02:01 host1 kernel: [<ffffffffa07a1990>] ?
gfs2_direct_IO+0x100/0x110 [gfs2]
Oct 24 13:02:01 host1 kernel: [<ffffffffa07a1760>] ?
gfs2_get_block_direct+0x0/0x20 [gfs2]
Oct 24 13:02:01 host1 kernel: [<ffffffffa07a18ec>] ?
gfs2_direct_IO+0x5c/0x110 [gfs2]
Oct 24 13:02:01 host1 kernel: [<ffffffff811160fb>] ?
generic_file_aio_read+0x6bb/0x700
Oct 24 13:02:01 host1 kernel: [<ffffffff8117afda>] ? do_sync_read+0xfa/0x140
Oct 24 13:02:01 host1 kernel: [<ffffffff81083a12>] ? send_signal+0x42/0x80
Oct 24 13:02:01 host1 kernel: [<ffffffff810920d0>] ?
autoremove_wake_function+0x0/0x40
Oct 24 13:02:01 host1 kernel: [<ffffffff81083e26>] ?
group_send_sig_info+0x56/0x70
Oct 24 13:02:01 host1 kernel: [<ffffffff8100bc6e>] ?
invalidate_interrupt1+0xe/0x20
Oct 24 13:02:01 host1 kernel: [<ffffffff8121fe8b>] ?
selinux_file_permission+0xfb/0x150
Oct 24 13:02:01 host1 kernel: [<ffffffff81213236>] ?
security_file_permission+0x16/0x20
Oct 24 13:02:01 host1 kernel: [<ffffffff8117b9e5>] ? vfs_read+0xb5/0x1a0
Oct 24 13:02:01 host1 kernel: [<ffffffff810d6b12>] ?
audit_syscall_entry+0x272/0x2a0
Oct 24 13:02:01 host1 kernel: [<ffffffff8117bd12>] ? sys_pread64+0x82/0xa0
Oct 24 13:02:01 host1 kernel: [<ffffffff8100b0f2>] ?
system_call_fastpath+0x16/0x1b
Oct 24 13:02:01 host1 kernel: ---[ end trace 4f331c7a492d6447 ]---
Oct 24 13:02:01 host1 kernel: ------------[ cut here ]------------
Oct 24 13:02:01 host1 kernel: WARNING: at fs/buffer.c:1199
__brelse+0x3a/0x40() (Tainted: G W --------------- )
Oct 24 13:02:01 host1 kernel: Hardware name: System x3630 M3 -[7377W03]-
Oct 24 13:02:01 host1 kernel: VFS: brelse: Trying to free free buffer
Oct 24 13:02:01 host1 kernel: Modules linked in: gfs2 dlm configfs ext3 jbd
nfs fscache ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE
xt_state nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc
cpufreq_ondemand acpi_cpufreq freq_table mperf bridge bonding 8021q garp stp
llc xt_CHECKSUM iptable_mangle ipt_REJECT iptable_filter iptable_nat nf_nat
nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables be2iscsi
iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio
ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp
libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan tun
kvm_intel kvm ses enclosure sg e1000e igb cdc_ether usbnet mii microcode
serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma dca
i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif pata_acpi
ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod
[last unloaded: configfs]
Oct 24 13:02:01 host1 kernel: Pid: 28581, comm: qemu-kvm Tainted: G W
--------------- 2.6.32-279.5.1.el6.x86_64 #1
Oct 24 13:02:01 host1 kernel: Call Trace:
Oct 24 13:02:01 host1 kernel: [<ffffffff8106b747>] ?
warn_slowpath_common+0x87/0xc0
Oct 24 13:02:01 host1 kernel: [<ffffffff8106b836>] ?
warn_slowpath_fmt+0x46/0x50
Oct 24 13:02:01 host1 kernel: [<ffffffff811ad48a>] ? __brelse+0x3a/0x40
Oct 24 13:02:01 host1 kernel: [<ffffffffa078bbc1>] ?
gfs2_block_map+0x1e1/0xf30 [gfs2]
Oct 24 13:02:01 host1 kernel: [<ffffffff81048fb5>] ?
gup_pte_range+0xe5/0x130
Oct 24 13:02:01 host1 kernel: [<ffffffff81049148>] ?
gup_pud_range+0x148/0x1d0
Oct 24 13:02:01 host1 kernel: [<ffffffffa07a1770>] ?
gfs2_get_block_direct+0x10/0x20 [gfs2]
Oct 24 13:02:01 host1 kernel: [<ffffffff811b689a>] ?
__blockdev_direct_IO_newtrunc+0x85a/0xb90
Oct 24 13:02:01 host1 kernel: [<ffffffff8112a965>] ?
pagevec_lookup_tag+0x25/0x40
Oct 24 13:02:01 host1 kernel: [<ffffffffa0799ca8>] ? do_promote+0x208/0x330
[gfs2]
Oct 24 13:02:01 host1 kernel: [<ffffffff811b6c2e>] ?
__blockdev_direct_IO+0x5e/0xd0
Oct 24 13:02:01 host1 kernel: [<ffffffffa07a1760>] ?
gfs2_get_block_direct+0x0/0x20 [gfs2]
Oct 24 13:02:01 host1 kernel: [<ffffffffa07a1990>] ?
gfs2_direct_IO+0x100/0x110 [gfs2]
Oct 24 13:02:01 host1 kernel: [<ffffffffa07a1760>] ?
gfs2_get_block_direct+0x0/0x20 [gfs2]
Oct 24 13:02:01 host1 kernel: [<ffffffffa07a18ec>] ?
gfs2_direct_IO+0x5c/0x110 [gfs2]
Oct 24 13:02:01 host1 kernel: [<ffffffff811160fb>] ?
generic_file_aio_read+0x6bb/0x700
Oct 24 13:02:01 host1 kernel: [<ffffffff810a4cf7>] ? futex_wait+0x227/0x380
Oct 24 13:02:01 host1 kernel: [<ffffffff81115a40>] ?
generic_file_aio_read+0x0/0x700
Oct 24 13:02:01 host1 kernel: [<ffffffff8117ad5b>] ?
do_sync_readv_writev+0xfb/0x140
Oct 24 13:02:01 host1 kernel: [<ffffffff810920d0>] ?
autoremove_wake_function+0x0/0x40
Oct 24 13:02:01 host1 kernel: [<ffffffff8121fe8b>] ?
selinux_file_permission+0xfb/0x150
Oct 24 13:02:01 host1 kernel: [<ffffffff81213236>] ?
security_file_permission+0x16/0x20
Oct 24 13:02:01 host1 kernel: [<ffffffff8117be06>] ?
do_readv_writev+0xd6/0x1f0
Oct 24 13:02:01 host1 kernel: [<ffffffff8117c133>] ? vfs_readv+0x43/0x60
Oct 24 13:02:01 host1 kernel: [<ffffffff8117c1f2>] ? sys_preadv+0xa2/0xc0
Oct 24 13:02:01 host1 kernel: [<ffffffff8100b0f2>] ?
system_call_fastpath+0x16/0x1b
Oct 24 13:02:01 host1 kernel: ---[ end trace 4f331c7a492d6448 ]---
Oct 24 13:02:02 host1 kernel: ------------[ cut here ]------------
Oct 24 13:02:02 host1 kernel: WARNING: at fs/buffer.c:1199
__brelse+0x3a/0x40() (Tainted: G W --------------- )
Oct 24 13:02:02 host1 kernel: Hardware name: System x3630 M3 -[7377W03]-
Oct 24 13:02:02 host1 kernel: VFS: brelse: Trying to free free buffer
Oct 24 13:02:02 host1 kernel: Modules linked in: gfs2 dlm configfs ext3 jbd
nfs fscache ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE
xt_state nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc
cpufreq_ondemand acpi_cpufreq freq_table mperf bridge bonding 8021q garp stp
llc xt_CHECKSUM iptable_mangle ipt_REJECT iptable_filter iptable_nat nf_nat
nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables be2iscsi
iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio
ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp
libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan tun
kvm_intel kvm ses enclosure sg e1000e igb cdc_ether usbnet mii microcode
serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma dca
i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif pata_acpi
ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod
[last unloaded: configfs]
Oct 24 13:02:02 host1 kernel: Pid: 28581, comm: qemu-kvm Tainted: G W
--------------- 2.6.32-279.5.1.el6.x86_64 #1
Oct 24 13:02:02 host1 kernel: Call Trace:
Oct 24 13:02:02 host1 kernel: [<ffffffff8106b747>] ?
warn_slowpath_common+0x87/0xc0
Oct 24 13:02:02 host1 kernel: [<ffffffff8106b836>] ?
warn_slowpath_fmt+0x46/0x50
Oct 24 13:02:02 host1 kernel: [<ffffffff8112a965>] ?
pagevec_lookup_tag+0x25/0x40
Oct 24 13:02:02 host1 kernel: [<ffffffff811ad48a>] ? __brelse+0x3a/0x40
Oct 24 13:02:02 host1 kernel: [<ffffffffa078bbc1>] ?
gfs2_block_map+0x1e1/0xf30 [gfs2]
Oct 24 13:02:02 host1 kernel: [<ffffffff81048fb5>] ?
gup_pte_range+0xe5/0x130
Oct 24 13:02:02 host1 kernel: [<ffffffff81049148>] ?
gup_pud_range+0x148/0x1d0
Oct 24 13:02:02 host1 kernel: [<ffffffffa07a1770>] ?
gfs2_get_block_direct+0x10/0x20 [gfs2]
Oct 24 13:02:02 host1 kernel: [<ffffffff811b689a>] ?
__blockdev_direct_IO_newtrunc+0x85a/0xb90
Oct 24 13:02:02 host1 kernel: [<ffffffff8112a965>] ?
pagevec_lookup_tag+0x25/0x40
Oct 24 13:02:02 host1 kernel: [<ffffffffa0799ca8>] ? do_promote+0x208/0x330
[gfs2]
Oct 24 13:02:02 host1 kernel: [<ffffffff811b6c2e>] ?
__blockdev_direct_IO+0x5e/0xd0
Oct 24 13:02:02 host1 kernel: [<ffffffffa07a1760>] ?
gfs2_get_block_direct+0x0/0x20 [gfs2]
Oct 24 13:02:02 host1 kernel: [<ffffffffa07a1990>] ?
gfs2_direct_IO+0x100/0x110 [gfs2]
Oct 24 13:02:02 host1 kernel: [<ffffffffa07a1760>] ?
gfs2_get_block_direct+0x0/0x20 [gfs2]
Oct 24 13:02:02 host1 kernel: [<ffffffffa07a18ec>] ?
gfs2_direct_IO+0x5c/0x110 [gfs2]
Oct 24 13:02:02 host1 kernel: [<ffffffff811160fb>] ?
generic_file_aio_read+0x6bb/0x700
Oct 24 13:02:02 host1 kernel: [<ffffffff8117afda>] ? do_sync_read+0xfa/0x140
Oct 24 13:02:02 host1 kernel: [<ffffffff81083a12>] ? send_signal+0x42/0x80
Oct 24 13:02:02 host1 kernel: [<ffffffff810920d0>] ?
autoremove_wake_function+0x0/0x40
Oct 24 13:02:02 host1 kernel: [<ffffffff81083e26>] ?
group_send_sig_info+0x56/0x70
Oct 24 13:02:02 host1 kernel: [<ffffffff8121fe8b>] ?
selinux_file_permission+0xfb/0x150
Oct 24 13:02:02 host1 kernel: [<ffffffff81213236>] ?
security_file_permission+0x16/0x20
Oct 24 13:02:02 host1 kernel: [<ffffffff8117b9e5>] ? vfs_read+0xb5/0x1a0
Oct 24 13:02:02 host1 kernel: [<ffffffff810d6b12>] ?
audit_syscall_entry+0x272/0x2a0
Oct 24 13:02:02 host1 kernel: [<ffffffff8117bd12>] ? sys_pread64+0x82/0xa0
Oct 24 13:02:02 host1 kernel: [<ffffffff8100b0f2>] ?
system_call_fastpath+0x16/0x1b
Oct 24 13:02:02 host1 kernel: ---[ end trace 4f331c7a492d6449 ]---
Oct 24 13:02:02 host1 abrt-dump-oops: Reported 3 kernel oopses to Abrt
Oct 24 13:02:02 host1 abrtd: Directory 'oops-2013-10-24-13:02:02-2922-2'
creation detected
Oct 24 13:02:02 host1 abrtd: Can't open file
'/var/spool/abrt/oops-2013-10-24-13:02:02-2922-2/uid': No such file or
directory
Oct 24 13:02:02 host1 kernel: ------------[ cut here ]------------
Oct 24 13:02:02 host1 kernel: WARNING: at fs/buffer.c:1199
__brelse+0x3a/0x40() (Tainted: G W --------------- )
Oct 24 13:02:02 host1 kernel: Hardware name: System x3630 M3 -[7377W03]-
Oct 24 13:02:02 host1 kernel: VFS: brelse: Trying to free free buffer
Oct 24 13:02:02 host1 kernel: Modules linked in: gfs2 dlm configfs ext3 jbd
nfs fscache ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE
xt_state nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc
cpufreq_ondemand acpi_cpufreq freq_table mperf bridge bonding 8021q garp stp
llc xt_CHECKSUM iptable_mangle ipt_REJECT iptable_filter iptable_nat nf_nat
nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables be2iscsi
iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio
ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp
libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan tun
kvm_intel kvm ses enclosure sg e1000e igb cdc_ether usbnet mii microcode
serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma dca
i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif pata_acpi
ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod
[last unloaded: configfs]
Oct 24 13:02:02 host1 kernel: Pid: 28581, comm: qemu-kvm Tainted: G W
--------------- 2.6.32-279.5.1.el6.x86_64 #1
Oct 24 13:02:02 host1 kernel: Call Trace:
Oct 24 13:02:02 host1 kernel: [<ffffffff8106b747>] ?
warn_slowpath_common+0x87/0xc0
Oct 24 13:02:02 host1 kernel: [<ffffffff8106b836>] ?
warn_slowpath_fmt+0x46/0x50
Oct 24 13:02:02 host1 kernel: [<ffffffff811b13ed>] ? bio_endio+0x1d/0x40
Oct 24 13:02:02 host1 kernel: [<ffffffff811ad48a>] ? __brelse+0x3a/0x40
Oct 24 13:02:02 host1 kernel: [<ffffffffa078bbc1>] ?
gfs2_block_map+0x1e1/0xf30 [gfs2]
Oct 24 13:02:02 host1 kernel: [<ffffffff81048fb5>] ?
gup_pte_range+0xe5/0x130
Oct 24 13:02:02 host1 kernel: [<ffffffff81049148>] ?
gup_pud_range+0x148/0x1d0
Oct 24 13:02:02 host1 kernel: [<ffffffffa07a1770>] ?
gfs2_get_block_direct+0x10/0x20 [gfs2]
Oct 24 13:02:02 host1 kernel: [<ffffffff811b689a>] ?
__blockdev_direct_IO_newtrunc+0x85a/0xb90
Oct 24 13:02:02 host1 kernel: [<ffffffff8112a965>] ?
pagevec_lookup_tag+0x25/0x40
Oct 24 13:02:02 host1 kernel: [<ffffffffa0799ca8>] ? do_promote+0x208/0x330
[gfs2]
Oct 24 13:02:02 host1 kernel: [<ffffffff811b6c2e>] ?
__blockdev_direct_IO+0x5e/0xd0
Oct 24 13:02:02 host1 kernel: [<ffffffffa07a1760>] ?
gfs2_get_block_direct+0x0/0x20 [gfs2]
Oct 24 13:02:02 host1 kernel: [<ffffffffa07a1990>] ?
gfs2_direct_IO+0x100/0x110 [gfs2]
Oct 24 13:02:02 host1 kernel: [<ffffffffa07a1760>] ?
gfs2_get_block_direct+0x0/0x20 [gfs2]
Oct 24 13:02:02 host1 kernel: [<ffffffffa07a18ec>] ?
gfs2_direct_IO+0x5c/0x110 [gfs2]
Oct 24 13:02:02 host1 kernel: [<ffffffff811160fb>] ?
generic_file_aio_read+0x6bb/0x700
Oct 24 13:02:02 host1 kernel: [<ffffffff810a4cf7>] ? futex_wait+0x227/0x380
Oct 24 13:02:02 host1 kernel: [<ffffffff81115a40>] ?
generic_file_aio_read+0x0/0x700
Oct 24 13:02:02 host1 kernel: [<ffffffff8117ad5b>] ?
do_sync_readv_writev+0xfb/0x140
Oct 24 13:02:02 host1 kernel: [<ffffffff810920d0>] ?
autoremove_wake_function+0x0/0x40
Oct 24 13:02:02 host1 kernel: [<ffffffff8121fe8b>] ?
selinux_file_permission+0xfb/0x150
Oct 24 13:02:02 host1 kernel: [<ffffffff81213236>] ?
security_file_permission+0x16/0x20
Oct 24 13:02:02 host1 kernel: [<ffffffff8117be06>] ?
do_readv_writev+0xd6/0x1f0
Oct 24 13:02:02 host1 kernel: [<ffffffff8117c133>] ? vfs_readv+0x43/0x60
Oct 24 13:02:02 host1 kernel: [<ffffffff8117c1f2>] ? sys_preadv+0xa2/0xc0
Oct 24 13:02:02 host1 kernel: [<ffffffff8100b0f2>] ?
system_call_fastpath+0x16/0x1b
Oct 24 13:02:02 host1 kernel: ---[ end trace 4f331c7a492d644a ]---
Oct 24 13:02:03 host1 kernel: ------------[ cut here ]------------
Oct 24 13:02:03 host1 kernel: WARNING: at fs/buffer.c:1199
__brelse+0x3a/0x40() (Tainted: G W --------------- )
Oct 24 13:02:03 host1 kernel: Hardware name: System x3630 M3 -[7377W03]-
Oct 24 13:02:03 host1 kernel: VFS: brelse: Trying to free free buffer
Oct 24 13:02:03 host1 kernel: Modules linked in: gfs2 dlm configfs ext3 jbd
nfs fscache ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE
xt_state nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc
cpufreq_ondemand acpi_cpufreq freq_table mperf bridge bonding 8021q garp stp
llc xt_CHECKSUM iptable_mangle ipt_REJECT iptable_filter iptable_nat nf_nat
nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables be2iscsi
iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio
ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp
libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan tun
kvm_intel kvm ses enclosure sg e1000e igb cdc_ether usbnet mii microcode
serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma dca
i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif pata_acpi
ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod
[last unloaded: configfs]
Oct 24 13:02:03 host1 kernel: Pid: 28581, comm: qemu-kvm Tainted: G W
--------------- 2.6.32-279.5.1.el6.x86_64 #1
Oct 24 13:02:03 host1 kernel: Call Trace:
Oct 24 13:02:03 host1 kernel: [<ffffffff8106b747>] ?
warn_slowpath_common+0x87/0xc0
Oct 24 13:02:03 host1 kernel: [<ffffffff8106b836>] ?
warn_slowpath_fmt+0x46/0x50
Oct 24 13:02:03 host1 kernel: [<ffffffff8112a965>] ?
pagevec_lookup_tag+0x25/0x40
Oct 24 13:02:03 host1 kernel: [<ffffffff811ad48a>] ? __brelse+0x3a/0x40
Oct 24 13:02:03 host1 kernel: [<ffffffffa078bbc1>] ?
gfs2_block_map+0x1e1/0xf30 [gfs2]
Oct 24 13:02:03 host1 kernel: [<ffffffff81048fb5>] ?
gup_pte_range+0xe5/0x130
Oct 24 13:02:03 host1 kernel: [<ffffffff81049148>] ?
gup_pud_range+0x148/0x1d0
Oct 24 13:02:03 host1 kernel: [<ffffffffa07a1770>] ?
gfs2_get_block_direct+0x10/0x20 [gfs2]
Oct 24 13:02:03 host1 kernel: [<ffffffff811b689a>] ?
__blockdev_direct_IO_newtrunc+0x85a/0xb90
Oct 24 13:02:03 host1 kernel: [<ffffffff8112a965>] ?
pagevec_lookup_tag+0x25/0x40
Oct 24 13:02:03 host1 kernel: [<ffffffffa0799ca8>] ? do_promote+0x208/0x330
[gfs2]
Oct 24 13:02:03 host1 kernel: [<ffffffff811b6c2e>] ?
__blockdev_direct_IO+0x5e/0xd0
Oct 24 13:02:03 host1 kernel: [<ffffffffa07a1760>] ?
gfs2_get_block_direct+0x0/0x20 [gfs2]
Oct 24 13:02:03 host1 kernel: [<ffffffffa07a1990>] ?
gfs2_direct_IO+0x100/0x110 [gfs2]
Oct 24 13:02:03 host1 kernel: [<ffffffffa07a1760>] ?
gfs2_get_block_direct+0x0/0x20 [gfs2]
Oct 24 13:02:03 host1 kernel: [<ffffffffa07a18ec>] ?
gfs2_direct_IO+0x5c/0x110 [gfs2]
Oct 24 13:02:03 host1 kernel: [<ffffffff811160fb>] ?
generic_file_aio_read+0x6bb/0x700
Oct 24 13:02:03 host1 kernel: [<ffffffff8117afda>] ? do_sync_read+0xfa/0x140
Oct 24 13:02:03 host1 kernel: [<ffffffff81083a12>] ? send_signal+0x42/0x80
Oct 24 13:02:03 host1 kernel: [<ffffffff810920d0>] ?
autoremove_wake_function+0x0/0x40
Oct 24 13:02:03 host1 kernel: [<ffffffff81083e26>] ?
group_send_sig_info+0x56/0x70
Oct 24 13:02:03 host1 kernel: [<ffffffff8121fe8b>] ?
selinux_file_permission+0xfb/0x150
Oct 24 13:02:03 host1 kernel: [<ffffffff81213236>] ?
security_file_permission+0x16/0x20
Oct 24 13:02:03 host1 kernel: [<ffffffff8117b9e5>] ? vfs_read+0xb5/0x1a0
Oct 24 13:02:03 host1 kernel: [<ffffffff810d6b12>] ?
audit_syscall_entry+0x272/0x2a0
Oct 24 13:02:03 host1 kernel: [<ffffffff8117bd12>] ? sys_pread64+0x82/0xa0
Oct 24 13:02:03 host1 kernel: [<ffffffff8100b0f2>] ?
system_call_fastpath+0x16/0x1b
Oct 24 13:02:03 host1 kernel: ---[ end trace 4f331c7a492d644b ]---
Oct 24 13:02:03 host1 kernel: ------------[ cut here ]------------
Oct 24 13:02:03 host1 kernel: WARNING: at fs/buffer.c:1199
__brelse+0x3a/0x40() (Tainted: G W --------------- )
Oct 24 13:02:03 host1 kernel: Hardware name: System x3630 M3 -[7377W03]-
Oct 24 13:02:03 host1 kernel: VFS: brelse: Trying to free free buffer
Oct 24 13:02:03 host1 kernel: Modules linked in: gfs2 dlm configfs ext3 jbd
nfs fscache ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE
xt_state nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc
cpufreq_ondemand acpi_cpufreq freq_table mperf bridge bonding 8021q garp stp
llc xt_CHECKSUM iptable_mangle ipt_REJECT iptable_filter iptable_nat nf_nat
nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables be2iscsi
iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio
ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp
libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan tun
kvm_intel kvm ses enclosure sg e1000e igb cdc_ether usbnet mii microcode
serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma dca
i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif pata_acpi
ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod
[last unloaded: configfs]
Oct 24 13:02:03 host1 kernel: Pid: 28581, comm: qemu-kvm Tainted: G W
--------------- 2.6.32-279.5.1.el6.x86_64 #1
Oct 24 13:02:03 host1 kernel: Call Trace:
Oct 24 13:02:03 host1 kernel: [<ffffffff8106b747>] ?
warn_slowpath_common+0x87/0xc0
Oct 24 13:02:03 host1 kernel: [<ffffffff8106b836>] ?
warn_slowpath_fmt+0x46/0x50
Oct 24 13:02:03 host1 kernel: [<ffffffff811b13ed>] ? bio_endio+0x1d/0x40
Oct 24 13:02:03 host1 kernel: [<ffffffff811ad48a>] ? __brelse+0x3a/0x40
Oct 24 13:02:03 host1 kernel: [<ffffffffa078bbc1>] ?
gfs2_block_map+0x1e1/0xf30 [gfs2]
Oct 24 13:02:03 host1 kernel: [<ffffffff81048fb5>] ?
gup_pte_range+0xe5/0x130
Oct 24 13:02:03 host1 kernel: [<ffffffff81049148>] ?
gup_pud_range+0x148/0x1d0
Oct 24 13:02:03 host1 kernel: [<ffffffffa07a1770>] ?
gfs2_get_block_direct+0x10/0x20 [gfs2]
Oct 24 13:02:03 host1 kernel: [<ffffffff811b689a>] ?
__blockdev_direct_IO_newtrunc+0x85a/0xb90
Oct 24 13:02:03 host1 kernel: [<ffffffff8112a965>] ?
pagevec_lookup_tag+0x25/0x40
Oct 24 13:02:03 host1 kernel: [<ffffffffa0799ca8>] ? do_promote+0x208/0x330
[gfs2]
Oct 24 13:02:03 host1 kernel: [<ffffffff811b6c2e>] ?
__blockdev_direct_IO+0x5e/0xd0
Oct 24 13:02:03 host1 kernel: [<ffffffffa07a1760>] ?
gfs2_get_block_direct+0x0/0x20 [gfs2]
Oct 24 13:02:03 host1 kernel: [<ffffffffa07a1990>] ?
gfs2_direct_IO+0x100/0x110 [gfs2]
Oct 24 13:02:03 host1 kernel: [<ffffffffa07a1760>] ?
gfs2_get_block_direct+0x0/0x20 [gfs2]
Oct 24 13:02:03 host1 kernel: [<ffffffffa07a18ec>] ?
gfs2_direct_IO+0x5c/0x110 [gfs2]
Oct 24 13:02:03 host1 kernel: [<ffffffff811160fb>] ?
generic_file_aio_read+0x6bb/0x700
Oct 24 13:02:03 host1 kernel: [<ffffffff810a4cf7>] ? futex_wait+0x227/0x380
Oct 24 13:02:03 host1 kernel: [<ffffffff81115a40>] ?
generic_file_aio_read+0x0/0x700
Oct 24 13:02:03 host1 kernel: [<ffffffff8117ad5b>] ?
do_sync_readv_writev+0xfb/0x140
Oct 24 13:02:03 host1 kernel: [<ffffffff810920d0>] ?
autoremove_wake_function+0x0/0x40
Oct 24 13:02:03 host1 kernel: [<ffffffff8121fe8b>] ?
selinux_file_permission+0xfb/0x150
Oct 24 13:02:03 host1 kernel: [<ffffffff81213236>] ?
security_file_permission+0x16/0x20
Oct 24 13:02:03 host1 kernel: [<ffffffff8117be06>] ?
do_readv_writev+0xd6/0x1f0
Oct 24 13:02:03 host1 kernel: [<ffffffff8117c133>] ? vfs_readv+0x43/0x60
Oct 24 13:02:03 host1 kernel: [<ffffffff8117c1f2>] ? sys_preadv+0xa2/0xc0
Oct 24 13:02:03 host1 kernel: [<ffffffff8100b0f2>] ?
system_call_fastpath+0x16/0x1b
Oct 24 13:02:03 host1 kernel: ---[ end trace 4f331c7a492d644c ]---
Oct 24 13:02:04 host1 kernel: ------------[ cut here ]------------
Oct 24 13:02:04 host1 kernel: WARNING: at fs/buffer.c:1199
__brelse+0x3a/0x40() (Tainted: G W --------------- )
Oct 24 13:02:04 host1 kernel: Hardware name: System x3630 M3 -[7377W03]-
Oct 24 13:02:04 host1 kernel: VFS: brelse: Trying to free free buffer
Oct 24 13:02:04 host1 kernel: Modules linked in: gfs2 dlm configfs ext3 jbd
nfs fscache ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE
xt_state nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc
cpufreq_ondemand acpi_cpufreq freq_table mperf bridge bonding 8021q garp stp
llc xt_CHECKSUM iptable_mangle ipt_REJECT iptable_filter iptable_nat nf_nat
nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables be2iscsi
iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio
ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp
libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan tun
kvm_intel kvm ses enclosure sg e1000e igb cdc_ether usbnet mii microcode
serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma dca
i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif pata_acpi
ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod
[last unloaded: configfs]
Oct 24 13:02:04 host1 kernel: Pid: 28581, comm: qemu-kvm Tainted: G W
--------------- 2.6.32-279.5.1.el6.x86_64 #1
Oct 24 13:02:04 host1 kernel: Call Trace:
Oct 24 13:02:04 host1 kernel: [<ffffffff8106b747>] ?
warn_slowpath_common+0x87/0xc0
Oct 24 13:02:04 host1 kernel: [<ffffffff8106b836>] ?
warn_slowpath_fmt+0x46/0x50
Oct 24 13:02:04 host1 kernel: [<ffffffff8112a965>] ?
pagevec_lookup_tag+0x25/0x40
Oct 24 13:02:04 host1 kernel: [<ffffffff811ad48a>] ? __brelse+0x3a/0x40
Oct 24 13:02:04 host1 kernel: [<ffffffffa078bbc1>] ?
gfs2_block_map+0x1e1/0xf30 [gfs2]
Oct 24 13:02:04 host1 kernel: [<ffffffff81048fb5>] ?
gup_pte_range+0xe5/0x130
Oct 24 13:02:04 host1 kernel: [<ffffffff81049148>] ?
gup_pud_range+0x148/0x1d0
Oct 24 13:02:04 host1 kernel: [<ffffffffa07a1770>] ?
gfs2_get_block_direct+0x10/0x20 [gfs2]
Oct 24 13:02:04 host1 kernel: [<ffffffff811b689a>] ?
__blockdev_direct_IO_newtrunc+0x85a/0xb90
Oct 24 13:02:04 host1 kernel: [<ffffffff8112a965>] ?
pagevec_lookup_tag+0x25/0x40
Oct 24 13:02:04 host1 kernel: [<ffffffffa0799ca8>] ? do_promote+0x208/0x330
[gfs2]
Oct 24 13:02:04 host1 kernel: [<ffffffff811b6c2e>] ?
__blockdev_direct_IO+0x5e/0xd0
Oct 24 13:02:04 host1 kernel: [<ffffffffa07a1760>] ?
gfs2_get_block_direct+0x0/0x20 [gfs2]
Oct 24 13:02:04 host1 kernel: [<ffffffffa07a1990>] ?
gfs2_direct_IO+0x100/0x110 [gfs2]
Oct 24 13:02:04 host1 kernel: [<ffffffffa07a1760>] ?
gfs2_get_block_direct+0x0/0x20 [gfs2]
Oct 24 13:02:04 host1 kernel: [<ffffffffa07a18ec>] ?
gfs2_direct_IO+0x5c/0x110 [gfs2]
Oct 24 13:02:04 host1 kernel: [<ffffffff811160fb>] ?
generic_file_aio_read+0x6bb/0x700
Oct 24 13:02:04 host1 kernel: [<ffffffff8117afda>] ? do_sync_read+0xfa/0x140
Oct 24 13:02:04 host1 kernel: [<ffffffff81083a12>] ? send_signal+0x42/0x80
Oct 24 13:02:04 host1 kernel: [<ffffffff810920d0>] ?
autoremove_wake_function+0x0/0x40
Oct 24 13:02:04 host1 kernel: [<ffffffff81083e26>] ?
group_send_sig_info+0x56/0x70
Oct 24 13:02:04 host1 kernel: [<ffffffff8121fe8b>] ?
selinux_file_permission+0xfb/0x150
Oct 24 13:02:04 host1 kernel: [<ffffffff81213236>] ?
security_file_permission+0x16/0x20
Oct 24 13:02:04 host1 kernel: [<ffffffff8117b9e5>] ? vfs_read+0xb5/0x1a0
Oct 24 13:02:04 host1 kernel: [<ffffffff810d6b12>] ?
audit_syscall_entry+0x272/0x2a0
Oct 24 13:02:04 host1 kernel: [<ffffffff8117bd12>] ? sys_pread64+0x82/0xa0
Oct 24 13:02:04 host1 kernel: [<ffffffff8100b0f2>] ?
system_call_fastpath+0x16/0x1b
Oct 24 13:02:04 host1 kernel: ---[ end trace 4f331c7a492d644d ]---
Oct 24 13:02:04 host1 abrt-dump-oops: Reported 4 kernel oopses to Abrt
Oct 24 13:02:04 host1 kernel: ------------[ cut here ]------------
Oct 24 13:02:04 host1 kernel: WARNING: at fs/buffer.c:1199
__brelse+0x3a/0x40() (Tainted: G W --------------- )
Oct 24 13:02:04 host1 kernel: Hardware name: System x3630 M3 -[7377W03]-
Oct 24 13:02:04 host1 kernel: VFS: brelse: Trying to free free buffer
Oct 24 13:02:04 host1 kernel: Modules linked in: gfs2 dlm configfs ext3 jbd
nfs fscache ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE
xt_state nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc
cpufreq_ondemand acpi_cpufreq freq_table mperf bridge bonding 8021q garp stp
llc xt_CHECKSUM iptable_mangle ipt_REJECT iptable_filter iptable_nat nf_nat
nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables be2iscsi
iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio
ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp
libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan tun
kvm_intel kvm ses enclosure sg e1000e igb cdc_ether usbnet mii microcode
serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma dca
i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif pata_acpi
ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod
[last unloaded: configfs]
Oct 24 13:02:04 host1 kernel: Pid: 28581, comm: qemu-kvm Tainted: G W
--------------- 2.6.32-279.5.1.el6.x86_64 #1
Oct 24 13:02:04 host1 kernel: Call Trace:
Oct 24 13:02:04 host1 kernel: [<ffffffff8106b747>] ?
warn_slowpath_common+0x87/0xc0
Oct 24 13:02:04 host1 kernel: [<ffffffff8106b836>] ?
warn_slowpath_fmt+0x46/0x50
Oct 24 13:02:04 host1 kernel: [<ffffffff811b13ed>] ? bio_endio+0x1d/0x40
Oct 24 13:02:04 host1 kernel: [<ffffffff811ad48a>] ? __brelse+0x3a/0x40
Oct 24 13:02:04 host1 kernel: [<ffffffffa078bbc1>] ?
gfs2_block_map+0x1e1/0xf30 [gfs2]
Oct 24 13:02:04 host1 kernel: [<ffffffff81048fb5>] ?
gup_pte_range+0xe5/0x130
Oct 24 13:02:04 host1 kernel: [<ffffffff81049148>] ?
gup_pud_range+0x148/0x1d0
Oct 24 13:02:04 host1 kernel: [<ffffffffa07a1770>] ?
gfs2_get_block_direct+0x10/0x20 [gfs2]
Oct 24 13:02:04 host1 kernel: [<ffffffff811b689a>] ?
__blockdev_direct_IO_newtrunc+0x85a/0xb90
Oct 24 13:02:04 host1 kernel: [<ffffffff8112a965>] ?
pagevec_lookup_tag+0x25/0x40
Oct 24 13:02:04 host1 kernel: [<ffffffffa0799ca8>] ? do_promote+0x208/0x330
[gfs2]
Oct 24 13:02:04 host1 kernel: [<ffffffff811b6c2e>] ?
__blockdev_direct_IO+0x5e/0xd0
Oct 24 13:02:04 host1 kernel: [<ffffffffa07a1760>] ?
gfs2_get_block_direct+0x0/0x20 [gfs2]
Oct 24 13:02:04 host1 kernel: [<ffffffffa07a1990>] ?
gfs2_direct_IO+0x100/0x110 [gfs2]
Oct 24 13:02:04 host1 kernel: [<ffffffffa07a1760>] ?
gfs2_get_block_direct+0x0/0x20 [gfs2]
Oct 24 13:02:04 host1 kernel: [<ffffffffa07a18ec>] ?
gfs2_direct_IO+0x5c/0x110 [gfs2]
Oct 24 13:02:04 host1 kernel: [<ffffffff811160fb>] ?
generic_file_aio_read+0x6bb/0x700
Oct 24 13:02:04 host1 kernel: [<ffffffff810a4cf7>] ? futex_wait+0x227/0x380
Oct 24 13:02:04 host1 kernel: [<ffffffff81115a40>] ?
generic_file_aio_read+0x0/0x700
Oct 24 13:02:04 host1 kernel: [<ffffffff8117ad5b>] ?
do_sync_readv_writev+0xfb/0x140
Oct 24 13:02:04 host1 kernel: [<ffffffff810920d0>] ?
autoremove_wake_function+0x0/0x40
Oct 24 13:02:04 host1 kernel: [<ffffffff8121fe8b>] ?
selinux_file_permission+0xfb/0x150
Oct 24 13:02:04 host1 kernel: [<ffffffff81213236>] ?
security_file_permission+0x16/0x20
Oct 24 13:02:04 host1 kernel: [<ffffffff8117be06>] ?
do_readv_writev+0xd6/0x1f0
Oct 24 13:02:04 host1 kernel: [<ffffffff8117c133>] ? vfs_readv+0x43/0x60
Oct 24 13:02:04 host1 kernel: [<ffffffff8117c1f2>] ? sys_preadv+0xa2/0xc0
Oct 24 13:02:04 host1 kernel: [<ffffffff8100b0f2>] ?
system_call_fastpath+0x16/0x1b
Oct 24 13:02:04 host1 kernel: ---[ end trace 4f331c7a492d644e ]---
Oct 24 13:02:05 host1 kernel: Buffer I/O error on device dm-0, logical block
434865136
Oct 24 13:02:05 host1 kernel: Buffer I/O error on device dm-0, logical block
434865136
Oct 24 13:02:05 host1 kernel: Buffer I/O error on device dm-0, logical block
434865150
Oct 24 13:02:05 host1 kernel: Buffer I/O error on device dm-0, logical block
434865150
Oct 24 13:02:05 host1 kernel: Buffer I/O error on device dm-0, logical block
0
Oct 24 13:02:05 host1 kernel: Buffer I/O error on device dm-0, logical block
0
Oct 24 13:02:05 host1 kernel: Buffer I/O error on device dm-0, logical block
1
Oct 24 13:02:05 host1 kernel: Buffer I/O error on device dm-0, logical block
434865151
Oct 24 13:02:05 host1 kernel: Buffer I/O error on device dm-0, logical block
1
Oct 24 13:02:05 host1 kernel: ------------[ cut here ]------------
Oct 24 13:02:05 host1 kernel: WARNING: at fs/buffer.c:1199
__brelse+0x3a/0x40() (Tainted: G W --------------- )
Oct 24 13:02:05 host1 kernel: Hardware name: System x3630 M3 -[7377W03]-
Oct 24 13:02:05 host1 kernel: VFS: brelse: Trying to free free buffer
Oct 24 13:02:05 host1 kernel: Modules linked in: gfs2 dlm configfs ext3 jbd
nfs fscache ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE
xt_state nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc
cpufreq_ondemand acpi_cpufreq freq_table mperf bridge bonding 8021q garp stp
llc xt_CHECKSUM iptable_mangle ipt_REJECT iptable_filter iptable_nat nf_nat
nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables be2iscsi
iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio
ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp
libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan tun
kvm_intel kvm ses enclosure sg e1000e igb cdc_ether usbnet mii microcode
serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma dca
i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif pata_acpi
ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod
[last unloaded: configfs]
Oct 24 13:02:05 host1 kernel: Pid: 28581, comm: qemu-kvm Tainted: G W
--------------- 2.6.32-279.5.1.el6.x86_64 #1
Oct 24 13:02:05 host1 kernel: Call Trace:
Oct 24 13:02:05 host1 kernel: [<ffffffff8106b747>] ?
warn_slowpath_common+0x87/0xc0
Oct 24 13:02:05 host1 kernel: [<ffffffff8106b836>] ?
warn_slowpath_fmt+0x46/0x50
Oct 24 13:02:05 host1 kernel: [<ffffffff811ad48a>] ? __brelse+0x3a/0x40
Oct 24 13:02:05 host1 kernel: [<ffffffffa078bbc1>] ?
gfs2_block_map+0x1e1/0xf30 [gfs2]
Oct 24 13:02:05 host1 kernel: [<ffffffff81048fb5>] ?
gup_pte_range+0xe5/0x130
Oct 24 13:02:05 host1 kernel: [<ffffffff81049148>] ?
gup_pud_range+0x148/0x1d0
Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1770>] ?
gfs2_get_block_direct+0x10/0x20 [gfs2]
Oct 24 13:02:05 host1 kernel: [<ffffffff811b689a>] ?
__blockdev_direct_IO_newtrunc+0x85a/0xb90
Oct 24 13:02:05 host1 kernel: [<ffffffff8112a965>] ?
pagevec_lookup_tag+0x25/0x40
Oct 24 13:02:05 host1 kernel: [<ffffffffa0799ca8>] ? do_promote+0x208/0x330
[gfs2]
Oct 24 13:02:05 host1 kernel: [<ffffffff811b6c2e>] ?
__blockdev_direct_IO+0x5e/0xd0
Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1760>] ?
gfs2_get_block_direct+0x0/0x20 [gfs2]
Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1990>] ?
gfs2_direct_IO+0x100/0x110 [gfs2]
Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1760>] ?
gfs2_get_block_direct+0x0/0x20 [gfs2]
Oct 24 13:02:05 host1 kernel: [<ffffffffa07a18ec>] ?
gfs2_direct_IO+0x5c/0x110 [gfs2]
Oct 24 13:02:05 host1 kernel: [<ffffffff811160fb>] ?
generic_file_aio_read+0x6bb/0x700
Oct 24 13:02:05 host1 kernel: [<ffffffff8117afda>] ? do_sync_read+0xfa/0x140
Oct 24 13:02:05 host1 kernel: [<ffffffff81083a12>] ? send_signal+0x42/0x80
Oct 24 13:02:05 host1 kernel: [<ffffffff810920d0>] ?
autoremove_wake_function+0x0/0x40
Oct 24 13:02:05 host1 kernel: [<ffffffff81083e26>] ?
group_send_sig_info+0x56/0x70
Oct 24 13:02:05 host1 kernel: [<ffffffff81083dd0>] ?
group_send_sig_info+0x0/0x70
Oct 24 13:02:05 host1 kernel: [<ffffffff8121fe8b>] ?
selinux_file_permission+0xfb/0x150
Oct 24 13:02:05 host1 kernel: [<ffffffff81213236>] ?
security_file_permission+0x16/0x20
Oct 24 13:02:05 host1 kernel: [<ffffffff8117b9e5>] ? vfs_read+0xb5/0x1a0
Oct 24 13:02:05 host1 kernel: [<ffffffff810d6b12>] ?
audit_syscall_entry+0x272/0x2a0
Oct 24 13:02:05 host1 kernel: [<ffffffff8117bd12>] ? sys_pread64+0x82/0xa0
Oct 24 13:02:05 host1 kernel: [<ffffffff8100b0f2>] ?
system_call_fastpath+0x16/0x1b
Oct 24 13:02:05 host1 kernel: ---[ end trace 4f331c7a492d644f ]---
Oct 24 13:02:05 host1 kernel: ------------[ cut here ]------------
Oct 24 13:02:05 host1 kernel: WARNING: at fs/buffer.c:1199
__brelse+0x3a/0x40() (Tainted: G W --------------- )
Oct 24 13:02:05 host1 kernel: Hardware name: System x3630 M3 -[7377W03]-
Oct 24 13:02:05 host1 kernel: VFS: brelse: Trying to free free buffer
Oct 24 13:02:05 host1 kernel: Modules linked in: gfs2 dlm configfs ext3 jbd
nfs fscache ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE
xt_state nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc
cpufreq_ondemand acpi_cpufreq freq_table mperf bridge bonding 8021q garp stp
llc xt_CHECKSUM iptable_mangle ipt_REJECT iptable_filter iptable_nat nf_nat
nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables be2iscsi
iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio
ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp
libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan tun
kvm_intel kvm ses enclosure sg e1000e igb cdc_ether usbnet mii microcode
serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma dca
i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif pata_acpi
ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod
[last unloaded: configfs]
Oct 24 13:02:05 host1 kernel: Pid: 28581, comm: qemu-kvm Tainted: G W
--------------- 2.6.32-279.5.1.el6.x86_64 #1
Oct 24 13:02:05 host1 kernel: Call Trace:
Oct 24 13:02:05 host1 kernel: [<ffffffff8106b747>] ?
warn_slowpath_common+0x87/0xc0
Oct 24 13:02:05 host1 kernel: [<ffffffff8106b836>] ?
warn_slowpath_fmt+0x46/0x50
Oct 24 13:02:05 host1 kernel: [<ffffffff811b13ed>] ? bio_endio+0x1d/0x40
Oct 24 13:02:05 host1 kernel: [<ffffffff811ad48a>] ? __brelse+0x3a/0x40
Oct 24 13:02:05 host1 kernel: [<ffffffffa078bbc1>] ?
gfs2_block_map+0x1e1/0xf30 [gfs2]
Oct 24 13:02:05 host1 kernel: [<ffffffff81048fb5>] ?
gup_pte_range+0xe5/0x130
Oct 24 13:02:05 host1 kernel: [<ffffffff81049148>] ?
gup_pud_range+0x148/0x1d0
Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1770>] ?
gfs2_get_block_direct+0x10/0x20 [gfs2]
Oct 24 13:02:05 host1 kernel: [<ffffffff811b689a>] ?
__blockdev_direct_IO_newtrunc+0x85a/0xb90
Oct 24 13:02:05 host1 kernel: [<ffffffff8112a965>] ?
pagevec_lookup_tag+0x25/0x40
Oct 24 13:02:05 host1 kernel: [<ffffffffa0799ca8>] ? do_promote+0x208/0x330
[gfs2]
Oct 24 13:02:05 host1 kernel: [<ffffffff811b6c2e>] ?
__blockdev_direct_IO+0x5e/0xd0
Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1760>] ?
gfs2_get_block_direct+0x0/0x20 [gfs2]
Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1990>] ?
gfs2_direct_IO+0x100/0x110 [gfs2]
Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1760>] ?
gfs2_get_block_direct+0x0/0x20 [gfs2]
Oct 24 13:02:05 host1 kernel: [<ffffffffa07a18ec>] ?
gfs2_direct_IO+0x5c/0x110 [gfs2]
Oct 24 13:02:05 host1 kernel: ------------[ cut here ]------------
Oct 24 13:02:05 host1 kernel: WARNING: at fs/buffer.c:1199
__brelse+0x3a/0x40() (Tainted: G W --------------- )
Oct 24 13:02:05 host1 kernel: Hardware name: System x3630 M3 -[7377W03]-
Oct 24 13:02:05 host1 kernel: VFS: brelse: Trying to free free buffer
Oct 24 13:02:05 host1 kernel: Modules linked in: gfs2 dlm configfs ext3 jbd
nfs fscache ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE
xt_state nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc
cpufreq_ondemand acpi_cpufreq freq_table mperf bridge bonding 8021q garp stp
llc xt_CHECKSUM iptable_mangle ipt_REJECT iptable_filter iptable_nat nf_nat
nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables be2iscsi
iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio
ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp
libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan tun
kvm_intel kvm ses enclosure sg e1000e igb cdc_ether usbnet mii microcode
serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma dca
i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif pata_acpi
ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod
[last unloaded: configfs]
Oct 24 13:02:05 host1 kernel: Pid: 3864, comm: qemu-kvm Tainted: G W
--------------- 2.6.32-279.5.1.el6.x86_64 #1
Oct 24 13:02:05 host1 kernel: Call Trace:
Oct 24 13:02:05 host1 kernel: [<ffffffff8106b747>] ?
warn_slowpath_common+0x87/0xc0
Oct 24 13:02:05 host1 kernel: [<ffffffff8106b836>] ?
warn_slowpath_fmt+0x46/0x50
Oct 24 13:02:05 host1 kernel: [<ffffffff81116a25>] ? mempool_free+0x95/0xa0
Oct 24 13:02:05 host1 kernel: [<ffffffff811ad48a>] ? __brelse+0x3a/0x40
Oct 24 13:02:05 host1 kernel: [<ffffffff811160fb>] ?
generic_file_aio_read+0x6bb/0x700
Oct 24 13:02:05 host1 kernel: [<ffffffff810a4cf7>] ? futex_wait+0x227/0x380
Oct 24 13:02:05 host1 kernel: [<ffffffffa078bbc1>] ?
gfs2_block_map+0x1e1/0xf30 [gfs2]
Oct 24 13:02:05 host1 kernel: [<ffffffff81048fb5>] ?
gup_pte_range+0xe5/0x130
Oct 24 13:02:05 host1 kernel: [<ffffffff81115a40>] ?
generic_file_aio_read+0x0/0x700
Oct 24 13:02:05 host1 kernel: [<ffffffff8117ad5b>] ?
do_sync_readv_writev+0xfb/0x140
Oct 24 13:02:05 host1 kernel: [<ffffffff81049148>] ?
gup_pud_range+0x148/0x1d0
Oct 24 13:02:05 host1 kernel: [<ffffffff810920d0>] ?
autoremove_wake_function+0x0/0x40
Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1770>] ?
gfs2_get_block_direct+0x10/0x20 [gfs2]
Oct 24 13:02:05 host1 kernel: [<ffffffff811b689a>] ?
__blockdev_direct_IO_newtrunc+0x85a/0xb90
Oct 24 13:02:05 host1 kernel: [<ffffffff8121fe8b>] ?
selinux_file_permission+0xfb/0x150
Oct 24 13:02:05 host1 kernel: [<ffffffffa0799ca8>] ? do_promote+0x208/0x330
[gfs2]
Oct 24 13:02:05 host1 kernel: [<ffffffff811b6c2e>] ?
__blockdev_direct_IO+0x5e/0xd0
Oct 24 13:02:05 host1 kernel: [<ffffffff81213236>] ?
security_file_permission+0x16/0x20
Oct 24 13:02:05 host1 kernel: [<ffffffff8117be06>] ?
do_readv_writev+0xd6/0x1f0
Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1760>] ?
gfs2_get_block_direct+0x0/0x20 [gfs2]
Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1990>] ?
gfs2_direct_IO+0x100/0x110 [gfs2]
Oct 24 13:02:05 host1 kernel: [<ffffffff8117c133>] ? vfs_readv+0x43/0x60
Oct 24 13:02:05 host1 kernel: [<ffffffff8117c1f2>] ? sys_preadv+0xa2/0xc0
Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1760>] ?
gfs2_get_block_direct+0x0/0x20 [gfs2]
Oct 24 13:02:05 host1 kernel: [<ffffffff8100b0f2>] ?
system_call_fastpath+0x16/0x1b
Oct 24 13:02:05 host1 kernel: [<ffffffff810a4cf7>] ? futex_wait+0x227/0x380
Oct 24 13:02:05 host1 kernel: ---[ end trace 4f331c7a492d6450 ]---
Oct 24 13:02:05 host1 kernel: [<ffffffffa07a18ec>] ?
gfs2_direct_IO+0x5c/0x110 [gfs2]
Oct 24 13:02:05 host1 kernel: [<ffffffff811160fb>] ?
generic_file_aio_read+0x6bb/0x700
Oct 24 13:02:05 host1 kernel: [<ffffffff8117afda>] ? do_sync_read+0xfa/0x140
Oct 24 13:02:05 host1 kernel: [<ffffffff81083a12>] ? send_signal+0x42/0x80
Oct 24 13:02:05 host1 kernel: [<ffffffff810920d0>] ?
autoremove_wake_function+0x0/0x40
Oct 24 13:02:05 host1 kernel: [<ffffffff81083e26>] ?
group_send_sig_info+0x56/0x70
Oct 24 13:02:05 host1 kernel: [<ffffffff8121fe8b>] ?
selinux_file_permission+0xfb/0x150
Oct 24 13:02:05 host1 kernel: [<ffffffff81213236>] ?
security_file_permission+0x16/0x20
Oct 24 13:02:05 host1 kernel: [<ffffffff8117b9e5>] ? vfs_read+0xb5/0x1a0
Oct 24 13:02:05 host1 kernel: [<ffffffff810d6b12>] ?
audit_syscall_entry+0x272/0x2a0
Oct 24 13:02:05 host1 kernel: [<ffffffff8117bd12>] ? sys_pread64+0x82/0xa0
Oct 24 13:02:05 host1 kernel: [<ffffffff8100b0f2>] ?
system_call_fastpath+0x16/0x1b
Oct 24 13:02:05 host1 kernel: ---[ end trace 4f331c7a492d6451 ]---
Oct 24 13:02:06 host1 kernel: GFS2: fsid=ckvm1_pod1:vg_psv4.0: fatal: I/O
error
Oct 24 13:02:06 host1 kernel: GFS2: fsid=ckvm1_pod1:vg_psv4.0: block =
14244
Oct 24 13:02:06 host1 kernel: GFS2: fsid=ckvm1_pod1:vg_psv4.0: function =
log_write_header, file = fs/gfs2/log.c, line = 616
Oct 24 13:02:06 host1 kernel: GFS2: fsid=ckvm1_pod1:vg_psv4.0: about to
withdraw this file system
Oct 24 13:02:06 host1 kernel: GFS2: fsid=ckvm1_pod1:vg_psv4.0: telling LM to
unmount
Oct 24 13:02:06 host1 kernel: GFS2: fsid=ckvm1_pod1:vg_psv4.0: withdrawn
Oct 24 13:02:06 host1 kernel: Pid: 4029, comm: parted Tainted: G W
--------------- 2.6.32-279.5.1.el6.x86_64 #1
Oct 24 13:02:06 host1 kernel: Call Trace:
Oct 24 13:02:06 host1 kernel: [<ffffffffa07b6062>] ?
gfs2_lm_withdraw+0x102/0x130 [gfs2]
Oct 24 13:02:06 host1 kernel: [<ffffffff814feb58>] ?
out_of_line_wait_on_bit+0x78/0x90
Oct 24 13:02:06 host1 kernel: [<ffffffff81092110>] ?
wake_bit_function+0x0/0x50
Oct 24 13:02:06 host1 kernel: [<ffffffffa07b60d0>] ?
gfs2_io_error_bh_i+0x40/0x50 [gfs2]
Oct 24 13:02:06 host1 kernel: [<ffffffff811ae0b6>] ?
__wait_on_buffer+0x26/0x30
Oct 24 13:02:06 host1 kernel: [<ffffffffa079e288>] ?
log_write_header+0x3a8/0x490 [gfs2]
Oct 24 13:02:06 host1 kernel: [<ffffffffa079e951>] ?
gfs2_log_flush+0x301/0x6f0 [gfs2]
Oct 24 13:02:06 host1 kernel: [<ffffffff811a48a9>] ?
sync_inodes_sb+0x149/0x190
Oct 24 13:02:06 host1 kernel: [<ffffffff810920d0>] ?
autoremove_wake_function+0x0/0x40
Oct 24 13:02:06 host1 kernel: [<ffffffffa07b24cc>] ? gfs2_sync_fs+0x2c/0x30
[gfs2]
Oct 24 13:02:06 host1 kernel: [<ffffffff811aa2e3>] ?
__sync_filesystem+0x53/0x90
Oct 24 13:02:06 host1 kernel: [<ffffffff811aa51b>] ?
sync_filesystem+0x4b/0x70
Oct 24 13:02:06 host1 kernel: [<ffffffff811b514e>] ? fsync_bdev+0x2e/0x70
Oct 24 13:02:06 host1 kernel: [<ffffffff8125ea9b>] ?
blkdev_ioctl+0x3ab/0x6e0
Oct 24 13:02:06 host1 kernel: [<ffffffff811b391c>] ? block_ioctl+0x3c/0x40
Oct 24 13:02:06 host1 kernel: [<ffffffff8118dff2>] ? vfs_ioctl+0x22/0xa0
Oct 24 13:02:06 host1 kernel: [<ffffffff8118e194>] ? do_vfs_ioctl+0x84/0x580
Oct 24 13:02:06 host1 kernel: [<ffffffff8118e711>] ? sys_ioctl+0x81/0xa0
Oct 24 13:02:06 host1 kernel: [<ffffffff8100b0f2>] ?
system_call_fastpath+0x16/0x1b
Oct 24 13:02:06 host1 abrt-dump-oops: Reported 4 kernel oopses to Abrt
Oct 24 13:02:29 host1 kernel: __ratelimit: 562 callbacks suppressed
Oct 24 13:02:29 host1 kernel: kvm: 23115: cpu0 unhandled wrmsr: 0x198 data 0
--- >8 ---
Now I have this "uncontrolled entry" and it doesn't allow me to start cman
again:
--- 8< ---
Oct 24 18:07:42 host1 dlm_controld[16712]: dlm_controld 3.0.12.1 started
Oct 24 18:07:43 host1 gfs_controld[16761]: gfs_controld 3.0.12.1 started
Oct 24 18:07:43 host1 gfs_controld[16761]: found uncontrolled gfs2 fs
ckvm1_pod1:vg_psv4
Oct 24 18:07:43 host1 gfs_controld[16761]: telling cman to remove nodeid 1
from cluster
Oct 24 18:07:43 host1 corosync[16628]: cman killed by node 1 because we were
killed by cman_tool or other application
Oct 24 18:07:43 host1 dlm_controld[16712]: cluster is down, exiting
Oct 24 18:07:43 host1 dlm_controld[16712]: daemon cpg_dispatch error 2
--- >8 ---
How to get rid of it? How to make node to "forget" about this gfs2?
Thank you!
-----Original Message-----
From: linux-cluster-bounces at redhat.com
[mailto:linux-cluster-bounces at redhat.com] On Behalf Of Digimer
Sent: Thursday, October 24, 2013 5:20 PM
To: linux clustering
Subject: Re: [Linux-cluster] uncontrolled entry
On 24/10/13 10:18, Vladimir Melnik wrote:
> Dear colleagues,
>
> I'll be very grateful for any clue or advice. I have a cluster
> (GFS2/DLM on iSCSI). Something happened today, so one node has stopped
> working in cluster and all operations with the filesystem was giving
"input/output errors".
> Other nodes are working okay.
>
> I unmounted the filesystem and stopped all services (gfs2, clvmd,
> cman, iscsi, iscsid), but I can't start cman again. That's what I have
> in
> log-files:
> fenced[2801]: found uncontrolled entry
> /sys/fs/gfs2/ckvm1_pod1:vg_psv4
> gfs_controld[2878]: found uncontrolled gfs2 fs ckvm1_pod1:vg_psv4
>
> Yes, I really have this "uncontrolled entry", how to get rid of it
> without rebooting the node?
>
> Thank you very much for any ideas and hints.
I've not seen that particular error before. Can you paste your cluster.conf
and the log files from the effected node, starting just before the event
occured? Please also look at the logs from other
node(s) at the same time and paste how they responded to the node failure.
--
Digimer
Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is
trapped in the mind of a person without access to education?
--
Linux-cluster mailing list
Linux-cluster at redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster
More information about the Linux-cluster
mailing list