[Linux-cluster] uncontrolled entry
Digimer
lists at alteeve.ca
Thu Oct 24 17:19:37 UTC 2013
Rebooting the one node should not impact GFS2 on the other nodes. I
don't know if you can kill -9 gfs2. In my experience, you need to
reboot. There is a decent chance it will fail to shut down, so I usually
handle cases like this by going to a good node and running 'fence_node
<victim>'. That will forcibly reboot it and tell the cluster to reap any
locks the hung node might still have.
Of course, I know little about your setup. If you need more
authoritative advice, I'd recommend opening a support ticket with Red Hat.
On 24/10/13 13:16, Vladimir Melnik wrote:
> Thanks a lot! Please, excuse me for ugly formatting.
>
> iSCSI works, but I'd like to restart GFS2 without rebooting the node as it
> works with other storages and it wouldn't be good to interrupt these
> processes.
>
> Any ideas on how to do that? Thank you.
>
>
> -----Original Message-----
> From: linux-cluster-bounces at redhat.com
> [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Digimer
> Sent: Thursday, October 24, 2013 6:35 PM
> To: linux clustering
> Subject: Re: [Linux-cluster] uncontrolled entry
>
> That was extremely hard to read, and the cluster.conf was edited to such
> a degree that it was no longer formatted correctly. That said, I think I
> can see enough to have an idea what happened.
>
> You do not manage the iSCSI connection or the gfs2 partition in the
> cluster, so the failure of the iSCSI connection and subsequent failure
> of the gfs2 partition were not a concern of the cluster. Corosync on
> that node was not interrupted, so the node was not fenced (that's only
> needed to put a node in an unknown state back into a known state).
>
> Have you resolved the iSCSI connection problem? As for gfs2, you stopped
> the cluster while it was still (trying to) run, so it's probably
> blocked. I'm not a GFS2 expert, so I can't speak to the internals of
> what is going on.
>
> If it was me, I would reboot the node, make sure iscsi was working after
> the reboot and then restart cman. I'd also put the storage components
> under the cluster's control, too.
>
> digimer
>
> On 24/10/13 11:11, Vladimir Melnik wrote:
>> Hi!
>>
>> Thank you. That's how clister.conf looks like:
>>
>> --- 8< ---
>> <?xml version="1.0"?>
>> <cluster name="ckvm1_pod1" config_version="5">
>> <clusternodes>
>> <clusternode name="bond0-101.host1.ckvm1.pod1.***.net" votes="1"
>> nodeid="1"><fence><method name="single"><device
>> name="host1_ipmi"/></method></fence></clusternode><clusternode
>> name="bond0-101.host2.ckvm1.pod1.***.net" votes="1"
>> nodeid="2"><fence><method name="single"><device
>> name="host2_ipmi"/></method></fence></clusternode><clusternode
>> name="bond0-101.host3.ckvm1.pod1.***.net" votes="1"
>> nodeid="3"><fence><method name="single"><device
>> name="host3_ipmi"/></method></fence></clusternode><clusternode
>> name="bond0-101.host4.ckvm1.pod1.***.net" votes="1"
>> nodeid="4"><fence><method name="single"><device
>> name="host4_ipmi"/></method></fence></clusternode></clusternodes>
>> <fencedevices>
>> <fencedevice name="host1_ipmi" agent="fence_ipmilan"
>> ipaddr="mc.host1.ckvm1.pod1.***.net" login="***"
> passwd="***"/><fencedevice
>> name="host2_ipmi" agent="fence_ipmilan"
> ipaddr="mc.host2.ckvm1.pod1.***.net"
>> login="***" passwd="***"/>="***"/><fencedevice name="host3_ipmi"
>> agent="fence_ipmilan" ipaddr="mc.host3.ckvm1.pod1.***.net" login="***"
>> passwd="***"/>="***"/><fencedevice name="host4_ipmi" agent="fence_ipmilan"
>> ipaddr="mc.host4.ckvm1.pod1.***.net" login="***"
>> passwd="***"/></fencedevices>
>> <rm>
>> <failoverdomains/>
>> <resources/>
>> </rm>
>> </cluster>
>> --- >8 ---
>>
>> That's what I've got in logg on the affected node:
>> --- 8< ---
>> Oct 24 13:01:57 host1 kernel: sd 10:0:0:0: [sdb] Synchronizing SCSI cache
>> Oct 24 13:01:57 host1 kernel: connection6:0: detected conn error (1020)
>> Oct 24 13:01:58 host1 kernel: scsi11 : iSCSI Initiator over TCP/IP
>> Oct 24 13:01:58 host1 kernel: scsi 11:0:0:0: Direct-Access OSNEXUS
>> QUANTASTOR 300 PQ: 0 ANSI: 6
>> Oct 24 13:01:58 host1 kernel: sd 11:0:0:0: [sdc] 3478923509 512-byte
> logical
>> blocks: (1.78 TB/1.61 TiB)
>> Oct 24 13:01:58 host1 kernel: sd 11:0:0:0: [sdc] 4096-byte physical blocks
>> Oct 24 13:01:58 host1 kernel: sd 11:0:0:0: [sdc] Write Protect is off
>> Oct 24 13:01:58 host1 kernel: sd 11:0:0:0: Attached scsi generic sg2 type
> 0
>> Oct 24 13:01:58 host1 kernel: sd 11:0:0:0: [sdc] Write cache: enabled,
> read
>> cache: enabled, supports DPO and FUA
>> Oct 24 13:01:58 host1 kernel: sdc: unknown partition table
>> Oct 24 13:01:58 host1 kernel: sd 11:0:0:0: [sdc] Attached SCSI disk
>> Oct 24 13:01:58 host1 iscsid: Connection6:0 to [target:
>> iqn.2009-10.com.osnexus:4eb54cf4-3f64da4ba5413ecc:PSV4, portal:
>> 192.168.***.***,3260] through [iface: default] is shutdown.
>> Oct 24 13:01:58 host1 iscsid: Connection7:0 to [target:
>> iqn.2009-10.com.osnexus:4eb54cf4-3f64da4ba5413ecc:PSV4, portal:
>> 192.168.***.***,3260] through [iface: default] is operational now
>> Oct 24 13:02:01 host1 kernel: ------------[ cut here ]------------
>> Oct 24 13:02:01 host1 kernel: WARNING: at fs/buffer.c:1199
>> __brelse+0x3a/0x40() (Not tainted)
>> Oct 24 13:02:01 host1 kernel: Hardware name: System x3630 M3 -[7377W03]-
>> Oct 24 13:02:01 host1 kernel: VFS: brelse: Trying to free free buffer
>> Oct 24 13:02:01 host1 kernel: Modules linked in: gfs2 dlm configfs ext3
> jbd
>> nfs fscache ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE
>> xt_state nfsd lockd nfs_acl auth_rpcgss
>> exportfs autofs4 sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf
>> bridge bonding 8021q garp stp llc xt_CHECKSUM iptable_mangle ipt_REJECT
>> iptable_filter iptable_nat nf_nat nf_conntrack_i
>> pv4 nf_conntrack nf_defrag_ipv4 ip_tables be2iscsi iscsi_boot_sysfs bnx2i
>> cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio ib_iser rdma_cm ib_cm
> iw_cm
>> ib_sa ib_mad ib_core ib_addr ipv6 iscsi_
>> tcp libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan
> tun
>> kvm_intel kvm ses enclosure sg e1000e igb cdc_ether usbnet mii microcode
>> serio_raw i2c_i801 i2c_core iTCO_wdt iTCO
>> _vendor_support ioatdma dca i7core_edac edac_core shpchp ext4 mbcache jbd2
>> sd_mod crc_t10dif pata_acpi ata_generic ata_piix megaraid_sas dm_mirror
>> dm_region_hash dm_log dm_mod [last unloaded:
>> configfs]
>> Oct 24 13:02:01 host1 kernel: Pid: 28581, comm: qemu-kvm Not tainted
>> 2.6.32-279.5.1.el6.x86_64 #1
>> Oct 24 13:02:01 host1 kernel: Call Trace:
>> Oct 24 13:02:01 host1 kernel: [<ffffffff8106b747>] ?
>> warn_slowpath_common+0x87/0xc0
>> Oct 24 13:02:01 host1 kernel: [<ffffffff8106b836>] ?
>> warn_slowpath_fmt+0x46/0x50
>> Oct 24 13:02:01 host1 kernel: [<ffffffff8112a965>] ?
>> pagevec_lookup_tag+0x25/0x40
>> Oct 24 13:02:01 host1 kernel: [<ffffffff811ad48a>] ? __brelse+0x3a/0x40
>> Oct 24 13:02:01 host1 kernel: [<ffffffffa078bbc1>] ?
>> gfs2_block_map+0x1e1/0xf30 [gfs2]
>> Oct 24 13:02:01 host1 kernel: [<ffffffff81048fb5>] ?
>> gup_pte_range+0xe5/0x130
>> Oct 24 13:02:01 host1 kernel: [<ffffffff81049148>] ?
>> gup_pud_range+0x148/0x1d0
>> Oct 24 13:02:01 host1 kernel: [<ffffffffa07a1770>] ?
>> gfs2_get_block_direct+0x10/0x20 [gfs2]
>> Oct 24 13:02:01 host1 kernel: [<ffffffff811b689a>] ?
>> __blockdev_direct_IO_newtrunc+0x85a/0xb90
>> Oct 24 13:02:01 host1 kernel: [<ffffffff8112a965>] ?
>> pagevec_lookup_tag+0x25/0x40
>> Oct 24 13:02:01 host1 kernel: [<ffffffffa0799ca8>] ?
> do_promote+0x208/0x330
>> [gfs2]
>> Oct 24 13:02:01 host1 kernel: [<ffffffff811b6c2e>] ?
>> __blockdev_direct_IO+0x5e/0xd0
>> Oct 24 13:02:01 host1 kernel: [<ffffffffa07a1760>] ?
>> gfs2_get_block_direct+0x0/0x20 [gfs2]
>> Oct 24 13:02:01 host1 kernel: [<ffffffffa07a1990>] ?
>> gfs2_direct_IO+0x100/0x110 [gfs2]
>> Oct 24 13:02:01 host1 kernel: [<ffffffffa07a1760>] ?
>> gfs2_get_block_direct+0x0/0x20 [gfs2]
>> Oct 24 13:02:01 host1 kernel: [<ffffffffa07a18ec>] ?
>> gfs2_direct_IO+0x5c/0x110 [gfs2]
>> Oct 24 13:02:01 host1 kernel: [<ffffffff811160fb>] ?
>> generic_file_aio_read+0x6bb/0x700
>> Oct 24 13:02:01 host1 kernel: [<ffffffff8117afda>] ?
> do_sync_read+0xfa/0x140
>> Oct 24 13:02:01 host1 kernel: [<ffffffff81083a12>] ? send_signal+0x42/0x80
>> Oct 24 13:02:01 host1 kernel: [<ffffffff810920d0>] ?
>> autoremove_wake_function+0x0/0x40
>> Oct 24 13:02:01 host1 kernel: [<ffffffff81083e26>] ?
>> group_send_sig_info+0x56/0x70
>> Oct 24 13:02:01 host1 kernel: [<ffffffff8100bc6e>] ?
>> invalidate_interrupt1+0xe/0x20
>> Oct 24 13:02:01 host1 kernel: [<ffffffff8121fe8b>] ?
>> selinux_file_permission+0xfb/0x150
>> Oct 24 13:02:01 host1 kernel: [<ffffffff81213236>] ?
>> security_file_permission+0x16/0x20
>> Oct 24 13:02:01 host1 kernel: [<ffffffff8117b9e5>] ? vfs_read+0xb5/0x1a0
>> Oct 24 13:02:01 host1 kernel: [<ffffffff810d6b12>] ?
>> audit_syscall_entry+0x272/0x2a0
>> Oct 24 13:02:01 host1 kernel: [<ffffffff8117bd12>] ? sys_pread64+0x82/0xa0
>> Oct 24 13:02:01 host1 kernel: [<ffffffff8100b0f2>] ?
>> system_call_fastpath+0x16/0x1b
>> Oct 24 13:02:01 host1 kernel: ---[ end trace 4f331c7a492d6447 ]---
>> Oct 24 13:02:01 host1 kernel: ------------[ cut here ]------------
>> Oct 24 13:02:01 host1 kernel: WARNING: at fs/buffer.c:1199
>> __brelse+0x3a/0x40() (Tainted: G W --------------- )
>> Oct 24 13:02:01 host1 kernel: Hardware name: System x3630 M3 -[7377W03]-
>> Oct 24 13:02:01 host1 kernel: VFS: brelse: Trying to free free buffer
>> Oct 24 13:02:01 host1 kernel: Modules linked in: gfs2 dlm configfs ext3
> jbd
>> nfs fscache ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE
>> xt_state nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc
>> cpufreq_ondemand acpi_cpufreq freq_table mperf bridge bonding 8021q garp
> stp
>> llc xt_CHECKSUM iptable_mangle ipt_REJECT iptable_filter iptable_nat
> nf_nat
>> nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables be2iscsi
>> iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio
>> ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp
>> libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan tun
>> kvm_intel kvm ses enclosure sg e1000e igb cdc_ether usbnet mii microcode
>> serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma dca
>> i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif pata_acpi
>> ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod
>> [last unloaded: configfs]
>> Oct 24 13:02:01 host1 kernel: Pid: 28581, comm: qemu-kvm Tainted: G
> W
>> --------------- 2.6.32-279.5.1.el6.x86_64 #1
>> Oct 24 13:02:01 host1 kernel: Call Trace:
>> Oct 24 13:02:01 host1 kernel: [<ffffffff8106b747>] ?
>> warn_slowpath_common+0x87/0xc0
>> Oct 24 13:02:01 host1 kernel: [<ffffffff8106b836>] ?
>> warn_slowpath_fmt+0x46/0x50
>> Oct 24 13:02:01 host1 kernel: [<ffffffff811ad48a>] ? __brelse+0x3a/0x40
>> Oct 24 13:02:01 host1 kernel: [<ffffffffa078bbc1>] ?
>> gfs2_block_map+0x1e1/0xf30 [gfs2]
>> Oct 24 13:02:01 host1 kernel: [<ffffffff81048fb5>] ?
>> gup_pte_range+0xe5/0x130
>> Oct 24 13:02:01 host1 kernel: [<ffffffff81049148>] ?
>> gup_pud_range+0x148/0x1d0
>> Oct 24 13:02:01 host1 kernel: [<ffffffffa07a1770>] ?
>> gfs2_get_block_direct+0x10/0x20 [gfs2]
>> Oct 24 13:02:01 host1 kernel: [<ffffffff811b689a>] ?
>> __blockdev_direct_IO_newtrunc+0x85a/0xb90
>> Oct 24 13:02:01 host1 kernel: [<ffffffff8112a965>] ?
>> pagevec_lookup_tag+0x25/0x40
>> Oct 24 13:02:01 host1 kernel: [<ffffffffa0799ca8>] ?
> do_promote+0x208/0x330
>> [gfs2]
>> Oct 24 13:02:01 host1 kernel: [<ffffffff811b6c2e>] ?
>> __blockdev_direct_IO+0x5e/0xd0
>> Oct 24 13:02:01 host1 kernel: [<ffffffffa07a1760>] ?
>> gfs2_get_block_direct+0x0/0x20 [gfs2]
>> Oct 24 13:02:01 host1 kernel: [<ffffffffa07a1990>] ?
>> gfs2_direct_IO+0x100/0x110 [gfs2]
>> Oct 24 13:02:01 host1 kernel: [<ffffffffa07a1760>] ?
>> gfs2_get_block_direct+0x0/0x20 [gfs2]
>> Oct 24 13:02:01 host1 kernel: [<ffffffffa07a18ec>] ?
>> gfs2_direct_IO+0x5c/0x110 [gfs2]
>> Oct 24 13:02:01 host1 kernel: [<ffffffff811160fb>] ?
>> generic_file_aio_read+0x6bb/0x700
>> Oct 24 13:02:01 host1 kernel: [<ffffffff810a4cf7>] ?
> futex_wait+0x227/0x380
>> Oct 24 13:02:01 host1 kernel: [<ffffffff81115a40>] ?
>> generic_file_aio_read+0x0/0x700
>> Oct 24 13:02:01 host1 kernel: [<ffffffff8117ad5b>] ?
>> do_sync_readv_writev+0xfb/0x140
>> Oct 24 13:02:01 host1 kernel: [<ffffffff810920d0>] ?
>> autoremove_wake_function+0x0/0x40
>> Oct 24 13:02:01 host1 kernel: [<ffffffff8121fe8b>] ?
>> selinux_file_permission+0xfb/0x150
>> Oct 24 13:02:01 host1 kernel: [<ffffffff81213236>] ?
>> security_file_permission+0x16/0x20
>> Oct 24 13:02:01 host1 kernel: [<ffffffff8117be06>] ?
>> do_readv_writev+0xd6/0x1f0
>> Oct 24 13:02:01 host1 kernel: [<ffffffff8117c133>] ? vfs_readv+0x43/0x60
>> Oct 24 13:02:01 host1 kernel: [<ffffffff8117c1f2>] ? sys_preadv+0xa2/0xc0
>> Oct 24 13:02:01 host1 kernel: [<ffffffff8100b0f2>] ?
>> system_call_fastpath+0x16/0x1b
>> Oct 24 13:02:01 host1 kernel: ---[ end trace 4f331c7a492d6448 ]---
>> Oct 24 13:02:02 host1 kernel: ------------[ cut here ]------------
>> Oct 24 13:02:02 host1 kernel: WARNING: at fs/buffer.c:1199
>> __brelse+0x3a/0x40() (Tainted: G W --------------- )
>> Oct 24 13:02:02 host1 kernel: Hardware name: System x3630 M3 -[7377W03]-
>> Oct 24 13:02:02 host1 kernel: VFS: brelse: Trying to free free buffer
>> Oct 24 13:02:02 host1 kernel: Modules linked in: gfs2 dlm configfs ext3
> jbd
>> nfs fscache ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE
>> xt_state nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc
>> cpufreq_ondemand acpi_cpufreq freq_table mperf bridge bonding 8021q garp
> stp
>> llc xt_CHECKSUM iptable_mangle ipt_REJECT iptable_filter iptable_nat
> nf_nat
>> nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables be2iscsi
>> iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio
>> ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp
>> libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan tun
>> kvm_intel kvm ses enclosure sg e1000e igb cdc_ether usbnet mii microcode
>> serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma dca
>> i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif pata_acpi
>> ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod
>> [last unloaded: configfs]
>> Oct 24 13:02:02 host1 kernel: Pid: 28581, comm: qemu-kvm Tainted: G
> W
>> --------------- 2.6.32-279.5.1.el6.x86_64 #1
>> Oct 24 13:02:02 host1 kernel: Call Trace:
>> Oct 24 13:02:02 host1 kernel: [<ffffffff8106b747>] ?
>> warn_slowpath_common+0x87/0xc0
>> Oct 24 13:02:02 host1 kernel: [<ffffffff8106b836>] ?
>> warn_slowpath_fmt+0x46/0x50
>> Oct 24 13:02:02 host1 kernel: [<ffffffff8112a965>] ?
>> pagevec_lookup_tag+0x25/0x40
>> Oct 24 13:02:02 host1 kernel: [<ffffffff811ad48a>] ? __brelse+0x3a/0x40
>> Oct 24 13:02:02 host1 kernel: [<ffffffffa078bbc1>] ?
>> gfs2_block_map+0x1e1/0xf30 [gfs2]
>> Oct 24 13:02:02 host1 kernel: [<ffffffff81048fb5>] ?
>> gup_pte_range+0xe5/0x130
>> Oct 24 13:02:02 host1 kernel: [<ffffffff81049148>] ?
>> gup_pud_range+0x148/0x1d0
>> Oct 24 13:02:02 host1 kernel: [<ffffffffa07a1770>] ?
>> gfs2_get_block_direct+0x10/0x20 [gfs2]
>> Oct 24 13:02:02 host1 kernel: [<ffffffff811b689a>] ?
>> __blockdev_direct_IO_newtrunc+0x85a/0xb90
>> Oct 24 13:02:02 host1 kernel: [<ffffffff8112a965>] ?
>> pagevec_lookup_tag+0x25/0x40
>> Oct 24 13:02:02 host1 kernel: [<ffffffffa0799ca8>] ?
> do_promote+0x208/0x330
>> [gfs2]
>> Oct 24 13:02:02 host1 kernel: [<ffffffff811b6c2e>] ?
>> __blockdev_direct_IO+0x5e/0xd0
>> Oct 24 13:02:02 host1 kernel: [<ffffffffa07a1760>] ?
>> gfs2_get_block_direct+0x0/0x20 [gfs2]
>> Oct 24 13:02:02 host1 kernel: [<ffffffffa07a1990>] ?
>> gfs2_direct_IO+0x100/0x110 [gfs2]
>> Oct 24 13:02:02 host1 kernel: [<ffffffffa07a1760>] ?
>> gfs2_get_block_direct+0x0/0x20 [gfs2]
>> Oct 24 13:02:02 host1 kernel: [<ffffffffa07a18ec>] ?
>> gfs2_direct_IO+0x5c/0x110 [gfs2]
>> Oct 24 13:02:02 host1 kernel: [<ffffffff811160fb>] ?
>> generic_file_aio_read+0x6bb/0x700
>> Oct 24 13:02:02 host1 kernel: [<ffffffff8117afda>] ?
> do_sync_read+0xfa/0x140
>> Oct 24 13:02:02 host1 kernel: [<ffffffff81083a12>] ? send_signal+0x42/0x80
>> Oct 24 13:02:02 host1 kernel: [<ffffffff810920d0>] ?
>> autoremove_wake_function+0x0/0x40
>> Oct 24 13:02:02 host1 kernel: [<ffffffff81083e26>] ?
>> group_send_sig_info+0x56/0x70
>> Oct 24 13:02:02 host1 kernel: [<ffffffff8121fe8b>] ?
>> selinux_file_permission+0xfb/0x150
>> Oct 24 13:02:02 host1 kernel: [<ffffffff81213236>] ?
>> security_file_permission+0x16/0x20
>> Oct 24 13:02:02 host1 kernel: [<ffffffff8117b9e5>] ? vfs_read+0xb5/0x1a0
>> Oct 24 13:02:02 host1 kernel: [<ffffffff810d6b12>] ?
>> audit_syscall_entry+0x272/0x2a0
>> Oct 24 13:02:02 host1 kernel: [<ffffffff8117bd12>] ? sys_pread64+0x82/0xa0
>> Oct 24 13:02:02 host1 kernel: [<ffffffff8100b0f2>] ?
>> system_call_fastpath+0x16/0x1b
>> Oct 24 13:02:02 host1 kernel: ---[ end trace 4f331c7a492d6449 ]---
>> Oct 24 13:02:02 host1 abrt-dump-oops: Reported 3 kernel oopses to Abrt
>> Oct 24 13:02:02 host1 abrtd: Directory 'oops-2013-10-24-13:02:02-2922-2'
>> creation detected
>> Oct 24 13:02:02 host1 abrtd: Can't open file
>> '/var/spool/abrt/oops-2013-10-24-13:02:02-2922-2/uid': No such file or
>> directory
>> Oct 24 13:02:02 host1 kernel: ------------[ cut here ]------------
>> Oct 24 13:02:02 host1 kernel: WARNING: at fs/buffer.c:1199
>> __brelse+0x3a/0x40() (Tainted: G W --------------- )
>> Oct 24 13:02:02 host1 kernel: Hardware name: System x3630 M3 -[7377W03]-
>> Oct 24 13:02:02 host1 kernel: VFS: brelse: Trying to free free buffer
>> Oct 24 13:02:02 host1 kernel: Modules linked in: gfs2 dlm configfs ext3
> jbd
>> nfs fscache ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE
>> xt_state nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc
>> cpufreq_ondemand acpi_cpufreq freq_table mperf bridge bonding 8021q garp
> stp
>> llc xt_CHECKSUM iptable_mangle ipt_REJECT iptable_filter iptable_nat
> nf_nat
>> nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables be2iscsi
>> iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio
>> ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp
>> libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan tun
>> kvm_intel kvm ses enclosure sg e1000e igb cdc_ether usbnet mii microcode
>> serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma dca
>> i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif pata_acpi
>> ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod
>> [last unloaded: configfs]
>> Oct 24 13:02:02 host1 kernel: Pid: 28581, comm: qemu-kvm Tainted: G
> W
>> --------------- 2.6.32-279.5.1.el6.x86_64 #1
>> Oct 24 13:02:02 host1 kernel: Call Trace:
>> Oct 24 13:02:02 host1 kernel: [<ffffffff8106b747>] ?
>> warn_slowpath_common+0x87/0xc0
>> Oct 24 13:02:02 host1 kernel: [<ffffffff8106b836>] ?
>> warn_slowpath_fmt+0x46/0x50
>> Oct 24 13:02:02 host1 kernel: [<ffffffff811b13ed>] ? bio_endio+0x1d/0x40
>> Oct 24 13:02:02 host1 kernel: [<ffffffff811ad48a>] ? __brelse+0x3a/0x40
>> Oct 24 13:02:02 host1 kernel: [<ffffffffa078bbc1>] ?
>> gfs2_block_map+0x1e1/0xf30 [gfs2]
>> Oct 24 13:02:02 host1 kernel: [<ffffffff81048fb5>] ?
>> gup_pte_range+0xe5/0x130
>> Oct 24 13:02:02 host1 kernel: [<ffffffff81049148>] ?
>> gup_pud_range+0x148/0x1d0
>> Oct 24 13:02:02 host1 kernel: [<ffffffffa07a1770>] ?
>> gfs2_get_block_direct+0x10/0x20 [gfs2]
>> Oct 24 13:02:02 host1 kernel: [<ffffffff811b689a>] ?
>> __blockdev_direct_IO_newtrunc+0x85a/0xb90
>> Oct 24 13:02:02 host1 kernel: [<ffffffff8112a965>] ?
>> pagevec_lookup_tag+0x25/0x40
>> Oct 24 13:02:02 host1 kernel: [<ffffffffa0799ca8>] ?
> do_promote+0x208/0x330
>> [gfs2]
>> Oct 24 13:02:02 host1 kernel: [<ffffffff811b6c2e>] ?
>> __blockdev_direct_IO+0x5e/0xd0
>> Oct 24 13:02:02 host1 kernel: [<ffffffffa07a1760>] ?
>> gfs2_get_block_direct+0x0/0x20 [gfs2]
>> Oct 24 13:02:02 host1 kernel: [<ffffffffa07a1990>] ?
>> gfs2_direct_IO+0x100/0x110 [gfs2]
>> Oct 24 13:02:02 host1 kernel: [<ffffffffa07a1760>] ?
>> gfs2_get_block_direct+0x0/0x20 [gfs2]
>> Oct 24 13:02:02 host1 kernel: [<ffffffffa07a18ec>] ?
>> gfs2_direct_IO+0x5c/0x110 [gfs2]
>> Oct 24 13:02:02 host1 kernel: [<ffffffff811160fb>] ?
>> generic_file_aio_read+0x6bb/0x700
>> Oct 24 13:02:02 host1 kernel: [<ffffffff810a4cf7>] ?
> futex_wait+0x227/0x380
>> Oct 24 13:02:02 host1 kernel: [<ffffffff81115a40>] ?
>> generic_file_aio_read+0x0/0x700
>> Oct 24 13:02:02 host1 kernel: [<ffffffff8117ad5b>] ?
>> do_sync_readv_writev+0xfb/0x140
>> Oct 24 13:02:02 host1 kernel: [<ffffffff810920d0>] ?
>> autoremove_wake_function+0x0/0x40
>> Oct 24 13:02:02 host1 kernel: [<ffffffff8121fe8b>] ?
>> selinux_file_permission+0xfb/0x150
>> Oct 24 13:02:02 host1 kernel: [<ffffffff81213236>] ?
>> security_file_permission+0x16/0x20
>> Oct 24 13:02:02 host1 kernel: [<ffffffff8117be06>] ?
>> do_readv_writev+0xd6/0x1f0
>> Oct 24 13:02:02 host1 kernel: [<ffffffff8117c133>] ? vfs_readv+0x43/0x60
>> Oct 24 13:02:02 host1 kernel: [<ffffffff8117c1f2>] ? sys_preadv+0xa2/0xc0
>> Oct 24 13:02:02 host1 kernel: [<ffffffff8100b0f2>] ?
>> system_call_fastpath+0x16/0x1b
>> Oct 24 13:02:02 host1 kernel: ---[ end trace 4f331c7a492d644a ]---
>> Oct 24 13:02:03 host1 kernel: ------------[ cut here ]------------
>> Oct 24 13:02:03 host1 kernel: WARNING: at fs/buffer.c:1199
>> __brelse+0x3a/0x40() (Tainted: G W --------------- )
>> Oct 24 13:02:03 host1 kernel: Hardware name: System x3630 M3 -[7377W03]-
>> Oct 24 13:02:03 host1 kernel: VFS: brelse: Trying to free free buffer
>> Oct 24 13:02:03 host1 kernel: Modules linked in: gfs2 dlm configfs ext3
> jbd
>> nfs fscache ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE
>> xt_state nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc
>> cpufreq_ondemand acpi_cpufreq freq_table mperf bridge bonding 8021q garp
> stp
>> llc xt_CHECKSUM iptable_mangle ipt_REJECT iptable_filter iptable_nat
> nf_nat
>> nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables be2iscsi
>> iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio
>> ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp
>> libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan tun
>> kvm_intel kvm ses enclosure sg e1000e igb cdc_ether usbnet mii microcode
>> serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma dca
>> i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif pata_acpi
>> ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod
>> [last unloaded: configfs]
>> Oct 24 13:02:03 host1 kernel: Pid: 28581, comm: qemu-kvm Tainted: G
> W
>> --------------- 2.6.32-279.5.1.el6.x86_64 #1
>> Oct 24 13:02:03 host1 kernel: Call Trace:
>> Oct 24 13:02:03 host1 kernel: [<ffffffff8106b747>] ?
>> warn_slowpath_common+0x87/0xc0
>> Oct 24 13:02:03 host1 kernel: [<ffffffff8106b836>] ?
>> warn_slowpath_fmt+0x46/0x50
>> Oct 24 13:02:03 host1 kernel: [<ffffffff8112a965>] ?
>> pagevec_lookup_tag+0x25/0x40
>> Oct 24 13:02:03 host1 kernel: [<ffffffff811ad48a>] ? __brelse+0x3a/0x40
>> Oct 24 13:02:03 host1 kernel: [<ffffffffa078bbc1>] ?
>> gfs2_block_map+0x1e1/0xf30 [gfs2]
>> Oct 24 13:02:03 host1 kernel: [<ffffffff81048fb5>] ?
>> gup_pte_range+0xe5/0x130
>> Oct 24 13:02:03 host1 kernel: [<ffffffff81049148>] ?
>> gup_pud_range+0x148/0x1d0
>> Oct 24 13:02:03 host1 kernel: [<ffffffffa07a1770>] ?
>> gfs2_get_block_direct+0x10/0x20 [gfs2]
>> Oct 24 13:02:03 host1 kernel: [<ffffffff811b689a>] ?
>> __blockdev_direct_IO_newtrunc+0x85a/0xb90
>> Oct 24 13:02:03 host1 kernel: [<ffffffff8112a965>] ?
>> pagevec_lookup_tag+0x25/0x40
>> Oct 24 13:02:03 host1 kernel: [<ffffffffa0799ca8>] ?
> do_promote+0x208/0x330
>> [gfs2]
>> Oct 24 13:02:03 host1 kernel: [<ffffffff811b6c2e>] ?
>> __blockdev_direct_IO+0x5e/0xd0
>> Oct 24 13:02:03 host1 kernel: [<ffffffffa07a1760>] ?
>> gfs2_get_block_direct+0x0/0x20 [gfs2]
>> Oct 24 13:02:03 host1 kernel: [<ffffffffa07a1990>] ?
>> gfs2_direct_IO+0x100/0x110 [gfs2]
>> Oct 24 13:02:03 host1 kernel: [<ffffffffa07a1760>] ?
>> gfs2_get_block_direct+0x0/0x20 [gfs2]
>> Oct 24 13:02:03 host1 kernel: [<ffffffffa07a18ec>] ?
>> gfs2_direct_IO+0x5c/0x110 [gfs2]
>> Oct 24 13:02:03 host1 kernel: [<ffffffff811160fb>] ?
>> generic_file_aio_read+0x6bb/0x700
>> Oct 24 13:02:03 host1 kernel: [<ffffffff8117afda>] ?
> do_sync_read+0xfa/0x140
>> Oct 24 13:02:03 host1 kernel: [<ffffffff81083a12>] ? send_signal+0x42/0x80
>> Oct 24 13:02:03 host1 kernel: [<ffffffff810920d0>] ?
>> autoremove_wake_function+0x0/0x40
>> Oct 24 13:02:03 host1 kernel: [<ffffffff81083e26>] ?
>> group_send_sig_info+0x56/0x70
>> Oct 24 13:02:03 host1 kernel: [<ffffffff8121fe8b>] ?
>> selinux_file_permission+0xfb/0x150
>> Oct 24 13:02:03 host1 kernel: [<ffffffff81213236>] ?
>> security_file_permission+0x16/0x20
>> Oct 24 13:02:03 host1 kernel: [<ffffffff8117b9e5>] ? vfs_read+0xb5/0x1a0
>> Oct 24 13:02:03 host1 kernel: [<ffffffff810d6b12>] ?
>> audit_syscall_entry+0x272/0x2a0
>> Oct 24 13:02:03 host1 kernel: [<ffffffff8117bd12>] ? sys_pread64+0x82/0xa0
>> Oct 24 13:02:03 host1 kernel: [<ffffffff8100b0f2>] ?
>> system_call_fastpath+0x16/0x1b
>> Oct 24 13:02:03 host1 kernel: ---[ end trace 4f331c7a492d644b ]---
>> Oct 24 13:02:03 host1 kernel: ------------[ cut here ]------------
>> Oct 24 13:02:03 host1 kernel: WARNING: at fs/buffer.c:1199
>> __brelse+0x3a/0x40() (Tainted: G W --------------- )
>> Oct 24 13:02:03 host1 kernel: Hardware name: System x3630 M3 -[7377W03]-
>> Oct 24 13:02:03 host1 kernel: VFS: brelse: Trying to free free buffer
>> Oct 24 13:02:03 host1 kernel: Modules linked in: gfs2 dlm configfs ext3
> jbd
>> nfs fscache ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE
>> xt_state nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc
>> cpufreq_ondemand acpi_cpufreq freq_table mperf bridge bonding 8021q garp
> stp
>> llc xt_CHECKSUM iptable_mangle ipt_REJECT iptable_filter iptable_nat
> nf_nat
>> nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables be2iscsi
>> iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio
>> ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp
>> libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan tun
>> kvm_intel kvm ses enclosure sg e1000e igb cdc_ether usbnet mii microcode
>> serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma dca
>> i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif pata_acpi
>> ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod
>> [last unloaded: configfs]
>> Oct 24 13:02:03 host1 kernel: Pid: 28581, comm: qemu-kvm Tainted: G
> W
>> --------------- 2.6.32-279.5.1.el6.x86_64 #1
>> Oct 24 13:02:03 host1 kernel: Call Trace:
>> Oct 24 13:02:03 host1 kernel: [<ffffffff8106b747>] ?
>> warn_slowpath_common+0x87/0xc0
>> Oct 24 13:02:03 host1 kernel: [<ffffffff8106b836>] ?
>> warn_slowpath_fmt+0x46/0x50
>> Oct 24 13:02:03 host1 kernel: [<ffffffff811b13ed>] ? bio_endio+0x1d/0x40
>> Oct 24 13:02:03 host1 kernel: [<ffffffff811ad48a>] ? __brelse+0x3a/0x40
>> Oct 24 13:02:03 host1 kernel: [<ffffffffa078bbc1>] ?
>> gfs2_block_map+0x1e1/0xf30 [gfs2]
>> Oct 24 13:02:03 host1 kernel: [<ffffffff81048fb5>] ?
>> gup_pte_range+0xe5/0x130
>> Oct 24 13:02:03 host1 kernel: [<ffffffff81049148>] ?
>> gup_pud_range+0x148/0x1d0
>> Oct 24 13:02:03 host1 kernel: [<ffffffffa07a1770>] ?
>> gfs2_get_block_direct+0x10/0x20 [gfs2]
>> Oct 24 13:02:03 host1 kernel: [<ffffffff811b689a>] ?
>> __blockdev_direct_IO_newtrunc+0x85a/0xb90
>> Oct 24 13:02:03 host1 kernel: [<ffffffff8112a965>] ?
>> pagevec_lookup_tag+0x25/0x40
>> Oct 24 13:02:03 host1 kernel: [<ffffffffa0799ca8>] ?
> do_promote+0x208/0x330
>> [gfs2]
>> Oct 24 13:02:03 host1 kernel: [<ffffffff811b6c2e>] ?
>> __blockdev_direct_IO+0x5e/0xd0
>> Oct 24 13:02:03 host1 kernel: [<ffffffffa07a1760>] ?
>> gfs2_get_block_direct+0x0/0x20 [gfs2]
>> Oct 24 13:02:03 host1 kernel: [<ffffffffa07a1990>] ?
>> gfs2_direct_IO+0x100/0x110 [gfs2]
>> Oct 24 13:02:03 host1 kernel: [<ffffffffa07a1760>] ?
>> gfs2_get_block_direct+0x0/0x20 [gfs2]
>> Oct 24 13:02:03 host1 kernel: [<ffffffffa07a18ec>] ?
>> gfs2_direct_IO+0x5c/0x110 [gfs2]
>> Oct 24 13:02:03 host1 kernel: [<ffffffff811160fb>] ?
>> generic_file_aio_read+0x6bb/0x700
>> Oct 24 13:02:03 host1 kernel: [<ffffffff810a4cf7>] ?
> futex_wait+0x227/0x380
>> Oct 24 13:02:03 host1 kernel: [<ffffffff81115a40>] ?
>> generic_file_aio_read+0x0/0x700
>> Oct 24 13:02:03 host1 kernel: [<ffffffff8117ad5b>] ?
>> do_sync_readv_writev+0xfb/0x140
>> Oct 24 13:02:03 host1 kernel: [<ffffffff810920d0>] ?
>> autoremove_wake_function+0x0/0x40
>> Oct 24 13:02:03 host1 kernel: [<ffffffff8121fe8b>] ?
>> selinux_file_permission+0xfb/0x150
>> Oct 24 13:02:03 host1 kernel: [<ffffffff81213236>] ?
>> security_file_permission+0x16/0x20
>> Oct 24 13:02:03 host1 kernel: [<ffffffff8117be06>] ?
>> do_readv_writev+0xd6/0x1f0
>> Oct 24 13:02:03 host1 kernel: [<ffffffff8117c133>] ? vfs_readv+0x43/0x60
>> Oct 24 13:02:03 host1 kernel: [<ffffffff8117c1f2>] ? sys_preadv+0xa2/0xc0
>> Oct 24 13:02:03 host1 kernel: [<ffffffff8100b0f2>] ?
>> system_call_fastpath+0x16/0x1b
>> Oct 24 13:02:03 host1 kernel: ---[ end trace 4f331c7a492d644c ]---
>> Oct 24 13:02:04 host1 kernel: ------------[ cut here ]------------
>> Oct 24 13:02:04 host1 kernel: WARNING: at fs/buffer.c:1199
>> __brelse+0x3a/0x40() (Tainted: G W --------------- )
>> Oct 24 13:02:04 host1 kernel: Hardware name: System x3630 M3 -[7377W03]-
>> Oct 24 13:02:04 host1 kernel: VFS: brelse: Trying to free free buffer
>> Oct 24 13:02:04 host1 kernel: Modules linked in: gfs2 dlm configfs ext3
> jbd
>> nfs fscache ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE
>> xt_state nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc
>> cpufreq_ondemand acpi_cpufreq freq_table mperf bridge bonding 8021q garp
> stp
>> llc xt_CHECKSUM iptable_mangle ipt_REJECT iptable_filter iptable_nat
> nf_nat
>> nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables be2iscsi
>> iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio
>> ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp
>> libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan tun
>> kvm_intel kvm ses enclosure sg e1000e igb cdc_ether usbnet mii microcode
>> serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma dca
>> i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif pata_acpi
>> ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod
>> [last unloaded: configfs]
>> Oct 24 13:02:04 host1 kernel: Pid: 28581, comm: qemu-kvm Tainted: G
> W
>> --------------- 2.6.32-279.5.1.el6.x86_64 #1
>> Oct 24 13:02:04 host1 kernel: Call Trace:
>> Oct 24 13:02:04 host1 kernel: [<ffffffff8106b747>] ?
>> warn_slowpath_common+0x87/0xc0
>> Oct 24 13:02:04 host1 kernel: [<ffffffff8106b836>] ?
>> warn_slowpath_fmt+0x46/0x50
>> Oct 24 13:02:04 host1 kernel: [<ffffffff8112a965>] ?
>> pagevec_lookup_tag+0x25/0x40
>> Oct 24 13:02:04 host1 kernel: [<ffffffff811ad48a>] ? __brelse+0x3a/0x40
>> Oct 24 13:02:04 host1 kernel: [<ffffffffa078bbc1>] ?
>> gfs2_block_map+0x1e1/0xf30 [gfs2]
>> Oct 24 13:02:04 host1 kernel: [<ffffffff81048fb5>] ?
>> gup_pte_range+0xe5/0x130
>> Oct 24 13:02:04 host1 kernel: [<ffffffff81049148>] ?
>> gup_pud_range+0x148/0x1d0
>> Oct 24 13:02:04 host1 kernel: [<ffffffffa07a1770>] ?
>> gfs2_get_block_direct+0x10/0x20 [gfs2]
>> Oct 24 13:02:04 host1 kernel: [<ffffffff811b689a>] ?
>> __blockdev_direct_IO_newtrunc+0x85a/0xb90
>> Oct 24 13:02:04 host1 kernel: [<ffffffff8112a965>] ?
>> pagevec_lookup_tag+0x25/0x40
>> Oct 24 13:02:04 host1 kernel: [<ffffffffa0799ca8>] ?
> do_promote+0x208/0x330
>> [gfs2]
>> Oct 24 13:02:04 host1 kernel: [<ffffffff811b6c2e>] ?
>> __blockdev_direct_IO+0x5e/0xd0
>> Oct 24 13:02:04 host1 kernel: [<ffffffffa07a1760>] ?
>> gfs2_get_block_direct+0x0/0x20 [gfs2]
>> Oct 24 13:02:04 host1 kernel: [<ffffffffa07a1990>] ?
>> gfs2_direct_IO+0x100/0x110 [gfs2]
>> Oct 24 13:02:04 host1 kernel: [<ffffffffa07a1760>] ?
>> gfs2_get_block_direct+0x0/0x20 [gfs2]
>> Oct 24 13:02:04 host1 kernel: [<ffffffffa07a18ec>] ?
>> gfs2_direct_IO+0x5c/0x110 [gfs2]
>> Oct 24 13:02:04 host1 kernel: [<ffffffff811160fb>] ?
>> generic_file_aio_read+0x6bb/0x700
>> Oct 24 13:02:04 host1 kernel: [<ffffffff8117afda>] ?
> do_sync_read+0xfa/0x140
>> Oct 24 13:02:04 host1 kernel: [<ffffffff81083a12>] ? send_signal+0x42/0x80
>> Oct 24 13:02:04 host1 kernel: [<ffffffff810920d0>] ?
>> autoremove_wake_function+0x0/0x40
>> Oct 24 13:02:04 host1 kernel: [<ffffffff81083e26>] ?
>> group_send_sig_info+0x56/0x70
>> Oct 24 13:02:04 host1 kernel: [<ffffffff8121fe8b>] ?
>> selinux_file_permission+0xfb/0x150
>> Oct 24 13:02:04 host1 kernel: [<ffffffff81213236>] ?
>> security_file_permission+0x16/0x20
>> Oct 24 13:02:04 host1 kernel: [<ffffffff8117b9e5>] ? vfs_read+0xb5/0x1a0
>> Oct 24 13:02:04 host1 kernel: [<ffffffff810d6b12>] ?
>> audit_syscall_entry+0x272/0x2a0
>> Oct 24 13:02:04 host1 kernel: [<ffffffff8117bd12>] ? sys_pread64+0x82/0xa0
>> Oct 24 13:02:04 host1 kernel: [<ffffffff8100b0f2>] ?
>> system_call_fastpath+0x16/0x1b
>> Oct 24 13:02:04 host1 kernel: ---[ end trace 4f331c7a492d644d ]---
>> Oct 24 13:02:04 host1 abrt-dump-oops: Reported 4 kernel oopses to Abrt
>> Oct 24 13:02:04 host1 kernel: ------------[ cut here ]------------
>> Oct 24 13:02:04 host1 kernel: WARNING: at fs/buffer.c:1199
>> __brelse+0x3a/0x40() (Tainted: G W --------------- )
>> Oct 24 13:02:04 host1 kernel: Hardware name: System x3630 M3 -[7377W03]-
>> Oct 24 13:02:04 host1 kernel: VFS: brelse: Trying to free free buffer
>> Oct 24 13:02:04 host1 kernel: Modules linked in: gfs2 dlm configfs ext3
> jbd
>> nfs fscache ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE
>> xt_state nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc
>> cpufreq_ondemand acpi_cpufreq freq_table mperf bridge bonding 8021q garp
> stp
>> llc xt_CHECKSUM iptable_mangle ipt_REJECT iptable_filter iptable_nat
> nf_nat
>> nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables be2iscsi
>> iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio
>> ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp
>> libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan tun
>> kvm_intel kvm ses enclosure sg e1000e igb cdc_ether usbnet mii microcode
>> serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma dca
>> i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif pata_acpi
>> ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod
>> [last unloaded: configfs]
>> Oct 24 13:02:04 host1 kernel: Pid: 28581, comm: qemu-kvm Tainted: G
> W
>> --------------- 2.6.32-279.5.1.el6.x86_64 #1
>> Oct 24 13:02:04 host1 kernel: Call Trace:
>> Oct 24 13:02:04 host1 kernel: [<ffffffff8106b747>] ?
>> warn_slowpath_common+0x87/0xc0
>> Oct 24 13:02:04 host1 kernel: [<ffffffff8106b836>] ?
>> warn_slowpath_fmt+0x46/0x50
>> Oct 24 13:02:04 host1 kernel: [<ffffffff811b13ed>] ? bio_endio+0x1d/0x40
>> Oct 24 13:02:04 host1 kernel: [<ffffffff811ad48a>] ? __brelse+0x3a/0x40
>> Oct 24 13:02:04 host1 kernel: [<ffffffffa078bbc1>] ?
>> gfs2_block_map+0x1e1/0xf30 [gfs2]
>> Oct 24 13:02:04 host1 kernel: [<ffffffff81048fb5>] ?
>> gup_pte_range+0xe5/0x130
>> Oct 24 13:02:04 host1 kernel: [<ffffffff81049148>] ?
>> gup_pud_range+0x148/0x1d0
>> Oct 24 13:02:04 host1 kernel: [<ffffffffa07a1770>] ?
>> gfs2_get_block_direct+0x10/0x20 [gfs2]
>> Oct 24 13:02:04 host1 kernel: [<ffffffff811b689a>] ?
>> __blockdev_direct_IO_newtrunc+0x85a/0xb90
>> Oct 24 13:02:04 host1 kernel: [<ffffffff8112a965>] ?
>> pagevec_lookup_tag+0x25/0x40
>> Oct 24 13:02:04 host1 kernel: [<ffffffffa0799ca8>] ?
> do_promote+0x208/0x330
>> [gfs2]
>> Oct 24 13:02:04 host1 kernel: [<ffffffff811b6c2e>] ?
>> __blockdev_direct_IO+0x5e/0xd0
>> Oct 24 13:02:04 host1 kernel: [<ffffffffa07a1760>] ?
>> gfs2_get_block_direct+0x0/0x20 [gfs2]
>> Oct 24 13:02:04 host1 kernel: [<ffffffffa07a1990>] ?
>> gfs2_direct_IO+0x100/0x110 [gfs2]
>> Oct 24 13:02:04 host1 kernel: [<ffffffffa07a1760>] ?
>> gfs2_get_block_direct+0x0/0x20 [gfs2]
>> Oct 24 13:02:04 host1 kernel: [<ffffffffa07a18ec>] ?
>> gfs2_direct_IO+0x5c/0x110 [gfs2]
>> Oct 24 13:02:04 host1 kernel: [<ffffffff811160fb>] ?
>> generic_file_aio_read+0x6bb/0x700
>> Oct 24 13:02:04 host1 kernel: [<ffffffff810a4cf7>] ?
> futex_wait+0x227/0x380
>> Oct 24 13:02:04 host1 kernel: [<ffffffff81115a40>] ?
>> generic_file_aio_read+0x0/0x700
>> Oct 24 13:02:04 host1 kernel: [<ffffffff8117ad5b>] ?
>> do_sync_readv_writev+0xfb/0x140
>> Oct 24 13:02:04 host1 kernel: [<ffffffff810920d0>] ?
>> autoremove_wake_function+0x0/0x40
>> Oct 24 13:02:04 host1 kernel: [<ffffffff8121fe8b>] ?
>> selinux_file_permission+0xfb/0x150
>> Oct 24 13:02:04 host1 kernel: [<ffffffff81213236>] ?
>> security_file_permission+0x16/0x20
>> Oct 24 13:02:04 host1 kernel: [<ffffffff8117be06>] ?
>> do_readv_writev+0xd6/0x1f0
>> Oct 24 13:02:04 host1 kernel: [<ffffffff8117c133>] ? vfs_readv+0x43/0x60
>> Oct 24 13:02:04 host1 kernel: [<ffffffff8117c1f2>] ? sys_preadv+0xa2/0xc0
>> Oct 24 13:02:04 host1 kernel: [<ffffffff8100b0f2>] ?
>> system_call_fastpath+0x16/0x1b
>> Oct 24 13:02:04 host1 kernel: ---[ end trace 4f331c7a492d644e ]---
>> Oct 24 13:02:05 host1 kernel: Buffer I/O error on device dm-0, logical
> block
>> 434865136
>> Oct 24 13:02:05 host1 kernel: Buffer I/O error on device dm-0, logical
> block
>> 434865136
>> Oct 24 13:02:05 host1 kernel: Buffer I/O error on device dm-0, logical
> block
>> 434865150
>> Oct 24 13:02:05 host1 kernel: Buffer I/O error on device dm-0, logical
> block
>> 434865150
>> Oct 24 13:02:05 host1 kernel: Buffer I/O error on device dm-0, logical
> block
>> 0
>> Oct 24 13:02:05 host1 kernel: Buffer I/O error on device dm-0, logical
> block
>> 0
>> Oct 24 13:02:05 host1 kernel: Buffer I/O error on device dm-0, logical
> block
>> 1
>> Oct 24 13:02:05 host1 kernel: Buffer I/O error on device dm-0, logical
> block
>> 434865151
>> Oct 24 13:02:05 host1 kernel: Buffer I/O error on device dm-0, logical
> block
>> 1
>> Oct 24 13:02:05 host1 kernel: ------------[ cut here ]------------
>> Oct 24 13:02:05 host1 kernel: WARNING: at fs/buffer.c:1199
>> __brelse+0x3a/0x40() (Tainted: G W --------------- )
>> Oct 24 13:02:05 host1 kernel: Hardware name: System x3630 M3 -[7377W03]-
>> Oct 24 13:02:05 host1 kernel: VFS: brelse: Trying to free free buffer
>> Oct 24 13:02:05 host1 kernel: Modules linked in: gfs2 dlm configfs ext3
> jbd
>> nfs fscache ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE
>> xt_state nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc
>> cpufreq_ondemand acpi_cpufreq freq_table mperf bridge bonding 8021q garp
> stp
>> llc xt_CHECKSUM iptable_mangle ipt_REJECT iptable_filter iptable_nat
> nf_nat
>> nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables be2iscsi
>> iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio
>> ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp
>> libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan tun
>> kvm_intel kvm ses enclosure sg e1000e igb cdc_ether usbnet mii microcode
>> serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma dca
>> i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif pata_acpi
>> ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod
>> [last unloaded: configfs]
>> Oct 24 13:02:05 host1 kernel: Pid: 28581, comm: qemu-kvm Tainted: G
> W
>> --------------- 2.6.32-279.5.1.el6.x86_64 #1
>> Oct 24 13:02:05 host1 kernel: Call Trace:
>> Oct 24 13:02:05 host1 kernel: [<ffffffff8106b747>] ?
>> warn_slowpath_common+0x87/0xc0
>> Oct 24 13:02:05 host1 kernel: [<ffffffff8106b836>] ?
>> warn_slowpath_fmt+0x46/0x50
>> Oct 24 13:02:05 host1 kernel: [<ffffffff811ad48a>] ? __brelse+0x3a/0x40
>> Oct 24 13:02:05 host1 kernel: [<ffffffffa078bbc1>] ?
>> gfs2_block_map+0x1e1/0xf30 [gfs2]
>> Oct 24 13:02:05 host1 kernel: [<ffffffff81048fb5>] ?
>> gup_pte_range+0xe5/0x130
>> Oct 24 13:02:05 host1 kernel: [<ffffffff81049148>] ?
>> gup_pud_range+0x148/0x1d0
>> Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1770>] ?
>> gfs2_get_block_direct+0x10/0x20 [gfs2]
>> Oct 24 13:02:05 host1 kernel: [<ffffffff811b689a>] ?
>> __blockdev_direct_IO_newtrunc+0x85a/0xb90
>> Oct 24 13:02:05 host1 kernel: [<ffffffff8112a965>] ?
>> pagevec_lookup_tag+0x25/0x40
>> Oct 24 13:02:05 host1 kernel: [<ffffffffa0799ca8>] ?
> do_promote+0x208/0x330
>> [gfs2]
>> Oct 24 13:02:05 host1 kernel: [<ffffffff811b6c2e>] ?
>> __blockdev_direct_IO+0x5e/0xd0
>> Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1760>] ?
>> gfs2_get_block_direct+0x0/0x20 [gfs2]
>> Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1990>] ?
>> gfs2_direct_IO+0x100/0x110 [gfs2]
>> Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1760>] ?
>> gfs2_get_block_direct+0x0/0x20 [gfs2]
>> Oct 24 13:02:05 host1 kernel: [<ffffffffa07a18ec>] ?
>> gfs2_direct_IO+0x5c/0x110 [gfs2]
>> Oct 24 13:02:05 host1 kernel: [<ffffffff811160fb>] ?
>> generic_file_aio_read+0x6bb/0x700
>> Oct 24 13:02:05 host1 kernel: [<ffffffff8117afda>] ?
> do_sync_read+0xfa/0x140
>> Oct 24 13:02:05 host1 kernel: [<ffffffff81083a12>] ? send_signal+0x42/0x80
>> Oct 24 13:02:05 host1 kernel: [<ffffffff810920d0>] ?
>> autoremove_wake_function+0x0/0x40
>> Oct 24 13:02:05 host1 kernel: [<ffffffff81083e26>] ?
>> group_send_sig_info+0x56/0x70
>> Oct 24 13:02:05 host1 kernel: [<ffffffff81083dd0>] ?
>> group_send_sig_info+0x0/0x70
>> Oct 24 13:02:05 host1 kernel: [<ffffffff8121fe8b>] ?
>> selinux_file_permission+0xfb/0x150
>> Oct 24 13:02:05 host1 kernel: [<ffffffff81213236>] ?
>> security_file_permission+0x16/0x20
>> Oct 24 13:02:05 host1 kernel: [<ffffffff8117b9e5>] ? vfs_read+0xb5/0x1a0
>> Oct 24 13:02:05 host1 kernel: [<ffffffff810d6b12>] ?
>> audit_syscall_entry+0x272/0x2a0
>> Oct 24 13:02:05 host1 kernel: [<ffffffff8117bd12>] ? sys_pread64+0x82/0xa0
>> Oct 24 13:02:05 host1 kernel: [<ffffffff8100b0f2>] ?
>> system_call_fastpath+0x16/0x1b
>> Oct 24 13:02:05 host1 kernel: ---[ end trace 4f331c7a492d644f ]---
>> Oct 24 13:02:05 host1 kernel: ------------[ cut here ]------------
>> Oct 24 13:02:05 host1 kernel: WARNING: at fs/buffer.c:1199
>> __brelse+0x3a/0x40() (Tainted: G W --------------- )
>> Oct 24 13:02:05 host1 kernel: Hardware name: System x3630 M3 -[7377W03]-
>> Oct 24 13:02:05 host1 kernel: VFS: brelse: Trying to free free buffer
>> Oct 24 13:02:05 host1 kernel: Modules linked in: gfs2 dlm configfs ext3
> jbd
>> nfs fscache ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE
>> xt_state nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc
>> cpufreq_ondemand acpi_cpufreq freq_table mperf bridge bonding 8021q garp
> stp
>> llc xt_CHECKSUM iptable_mangle ipt_REJECT iptable_filter iptable_nat
> nf_nat
>> nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables be2iscsi
>> iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio
>> ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp
>> libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan tun
>> kvm_intel kvm ses enclosure sg e1000e igb cdc_ether usbnet mii microcode
>> serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma dca
>> i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif pata_acpi
>> ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod
>> [last unloaded: configfs]
>> Oct 24 13:02:05 host1 kernel: Pid: 28581, comm: qemu-kvm Tainted: G
> W
>> --------------- 2.6.32-279.5.1.el6.x86_64 #1
>> Oct 24 13:02:05 host1 kernel: Call Trace:
>> Oct 24 13:02:05 host1 kernel: [<ffffffff8106b747>] ?
>> warn_slowpath_common+0x87/0xc0
>> Oct 24 13:02:05 host1 kernel: [<ffffffff8106b836>] ?
>> warn_slowpath_fmt+0x46/0x50
>> Oct 24 13:02:05 host1 kernel: [<ffffffff811b13ed>] ? bio_endio+0x1d/0x40
>> Oct 24 13:02:05 host1 kernel: [<ffffffff811ad48a>] ? __brelse+0x3a/0x40
>> Oct 24 13:02:05 host1 kernel: [<ffffffffa078bbc1>] ?
>> gfs2_block_map+0x1e1/0xf30 [gfs2]
>> Oct 24 13:02:05 host1 kernel: [<ffffffff81048fb5>] ?
>> gup_pte_range+0xe5/0x130
>> Oct 24 13:02:05 host1 kernel: [<ffffffff81049148>] ?
>> gup_pud_range+0x148/0x1d0
>> Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1770>] ?
>> gfs2_get_block_direct+0x10/0x20 [gfs2]
>> Oct 24 13:02:05 host1 kernel: [<ffffffff811b689a>] ?
>> __blockdev_direct_IO_newtrunc+0x85a/0xb90
>> Oct 24 13:02:05 host1 kernel: [<ffffffff8112a965>] ?
>> pagevec_lookup_tag+0x25/0x40
>> Oct 24 13:02:05 host1 kernel: [<ffffffffa0799ca8>] ?
> do_promote+0x208/0x330
>> [gfs2]
>> Oct 24 13:02:05 host1 kernel: [<ffffffff811b6c2e>] ?
>> __blockdev_direct_IO+0x5e/0xd0
>> Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1760>] ?
>> gfs2_get_block_direct+0x0/0x20 [gfs2]
>> Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1990>] ?
>> gfs2_direct_IO+0x100/0x110 [gfs2]
>> Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1760>] ?
>> gfs2_get_block_direct+0x0/0x20 [gfs2]
>> Oct 24 13:02:05 host1 kernel: [<ffffffffa07a18ec>] ?
>> gfs2_direct_IO+0x5c/0x110 [gfs2]
>> Oct 24 13:02:05 host1 kernel: ------------[ cut here ]------------
>> Oct 24 13:02:05 host1 kernel: WARNING: at fs/buffer.c:1199
>> __brelse+0x3a/0x40() (Tainted: G W --------------- )
>> Oct 24 13:02:05 host1 kernel: Hardware name: System x3630 M3 -[7377W03]-
>> Oct 24 13:02:05 host1 kernel: VFS: brelse: Trying to free free buffer
>> Oct 24 13:02:05 host1 kernel: Modules linked in: gfs2 dlm configfs ext3
> jbd
>> nfs fscache ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE
>> xt_state nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc
>> cpufreq_ondemand acpi_cpufreq freq_table mperf bridge bonding 8021q garp
> stp
>> llc xt_CHECKSUM iptable_mangle ipt_REJECT iptable_filter iptable_nat
> nf_nat
>> nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables be2iscsi
>> iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio
>> ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp
>> libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan tun
>> kvm_intel kvm ses enclosure sg e1000e igb cdc_ether usbnet mii microcode
>> serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma dca
>> i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif pata_acpi
>> ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod
>> [last unloaded: configfs]
>> Oct 24 13:02:05 host1 kernel: Pid: 3864, comm: qemu-kvm Tainted: G
> W
>> --------------- 2.6.32-279.5.1.el6.x86_64 #1
>> Oct 24 13:02:05 host1 kernel: Call Trace:
>> Oct 24 13:02:05 host1 kernel: [<ffffffff8106b747>] ?
>> warn_slowpath_common+0x87/0xc0
>> Oct 24 13:02:05 host1 kernel: [<ffffffff8106b836>] ?
>> warn_slowpath_fmt+0x46/0x50
>> Oct 24 13:02:05 host1 kernel: [<ffffffff81116a25>] ?
> mempool_free+0x95/0xa0
>> Oct 24 13:02:05 host1 kernel: [<ffffffff811ad48a>] ? __brelse+0x3a/0x40
>> Oct 24 13:02:05 host1 kernel: [<ffffffff811160fb>] ?
>> generic_file_aio_read+0x6bb/0x700
>> Oct 24 13:02:05 host1 kernel: [<ffffffff810a4cf7>] ?
> futex_wait+0x227/0x380
>> Oct 24 13:02:05 host1 kernel: [<ffffffffa078bbc1>] ?
>> gfs2_block_map+0x1e1/0xf30 [gfs2]
>> Oct 24 13:02:05 host1 kernel: [<ffffffff81048fb5>] ?
>> gup_pte_range+0xe5/0x130
>> Oct 24 13:02:05 host1 kernel: [<ffffffff81115a40>] ?
>> generic_file_aio_read+0x0/0x700
>> Oct 24 13:02:05 host1 kernel: [<ffffffff8117ad5b>] ?
>> do_sync_readv_writev+0xfb/0x140
>> Oct 24 13:02:05 host1 kernel: [<ffffffff81049148>] ?
>> gup_pud_range+0x148/0x1d0
>> Oct 24 13:02:05 host1 kernel: [<ffffffff810920d0>] ?
>> autoremove_wake_function+0x0/0x40
>> Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1770>] ?
>> gfs2_get_block_direct+0x10/0x20 [gfs2]
>> Oct 24 13:02:05 host1 kernel: [<ffffffff811b689a>] ?
>> __blockdev_direct_IO_newtrunc+0x85a/0xb90
>> Oct 24 13:02:05 host1 kernel: [<ffffffff8121fe8b>] ?
>> selinux_file_permission+0xfb/0x150
>> Oct 24 13:02:05 host1 kernel: [<ffffffffa0799ca8>] ?
> do_promote+0x208/0x330
>> [gfs2]
>> Oct 24 13:02:05 host1 kernel: [<ffffffff811b6c2e>] ?
>> __blockdev_direct_IO+0x5e/0xd0
>> Oct 24 13:02:05 host1 kernel: [<ffffffff81213236>] ?
>> security_file_permission+0x16/0x20
>> Oct 24 13:02:05 host1 kernel: [<ffffffff8117be06>] ?
>> do_readv_writev+0xd6/0x1f0
>> Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1760>] ?
>> gfs2_get_block_direct+0x0/0x20 [gfs2]
>> Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1990>] ?
>> gfs2_direct_IO+0x100/0x110 [gfs2]
>> Oct 24 13:02:05 host1 kernel: [<ffffffff8117c133>] ? vfs_readv+0x43/0x60
>> Oct 24 13:02:05 host1 kernel: [<ffffffff8117c1f2>] ? sys_preadv+0xa2/0xc0
>> Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1760>] ?
>> gfs2_get_block_direct+0x0/0x20 [gfs2]
>> Oct 24 13:02:05 host1 kernel: [<ffffffff8100b0f2>] ?
>> system_call_fastpath+0x16/0x1b
>> Oct 24 13:02:05 host1 kernel: [<ffffffff810a4cf7>] ?
> futex_wait+0x227/0x380
>> Oct 24 13:02:05 host1 kernel: ---[ end trace 4f331c7a492d6450 ]---
>> Oct 24 13:02:05 host1 kernel: [<ffffffffa07a18ec>] ?
>> gfs2_direct_IO+0x5c/0x110 [gfs2]
>> Oct 24 13:02:05 host1 kernel: [<ffffffff811160fb>] ?
>> generic_file_aio_read+0x6bb/0x700
>> Oct 24 13:02:05 host1 kernel: [<ffffffff8117afda>] ?
> do_sync_read+0xfa/0x140
>> Oct 24 13:02:05 host1 kernel: [<ffffffff81083a12>] ? send_signal+0x42/0x80
>> Oct 24 13:02:05 host1 kernel: [<ffffffff810920d0>] ?
>> autoremove_wake_function+0x0/0x40
>> Oct 24 13:02:05 host1 kernel: [<ffffffff81083e26>] ?
>> group_send_sig_info+0x56/0x70
>> Oct 24 13:02:05 host1 kernel: [<ffffffff8121fe8b>] ?
>> selinux_file_permission+0xfb/0x150
>> Oct 24 13:02:05 host1 kernel: [<ffffffff81213236>] ?
>> security_file_permission+0x16/0x20
>> Oct 24 13:02:05 host1 kernel: [<ffffffff8117b9e5>] ? vfs_read+0xb5/0x1a0
>> Oct 24 13:02:05 host1 kernel: [<ffffffff810d6b12>] ?
>> audit_syscall_entry+0x272/0x2a0
>> Oct 24 13:02:05 host1 kernel: [<ffffffff8117bd12>] ? sys_pread64+0x82/0xa0
>> Oct 24 13:02:05 host1 kernel: [<ffffffff8100b0f2>] ?
>> system_call_fastpath+0x16/0x1b
>> Oct 24 13:02:05 host1 kernel: ---[ end trace 4f331c7a492d6451 ]---
>> Oct 24 13:02:06 host1 kernel: GFS2: fsid=ckvm1_pod1:vg_psv4.0: fatal: I/O
>> error
>> Oct 24 13:02:06 host1 kernel: GFS2: fsid=ckvm1_pod1:vg_psv4.0: block =
>> 14244
>> Oct 24 13:02:06 host1 kernel: GFS2: fsid=ckvm1_pod1:vg_psv4.0: function
> =
>> log_write_header, file = fs/gfs2/log.c, line = 616
>> Oct 24 13:02:06 host1 kernel: GFS2: fsid=ckvm1_pod1:vg_psv4.0: about to
>> withdraw this file system
>> Oct 24 13:02:06 host1 kernel: GFS2: fsid=ckvm1_pod1:vg_psv4.0: telling LM
> to
>> unmount
>> Oct 24 13:02:06 host1 kernel: GFS2: fsid=ckvm1_pod1:vg_psv4.0: withdrawn
>> Oct 24 13:02:06 host1 kernel: Pid: 4029, comm: parted Tainted: G W
>> --------------- 2.6.32-279.5.1.el6.x86_64 #1
>> Oct 24 13:02:06 host1 kernel: Call Trace:
>> Oct 24 13:02:06 host1 kernel: [<ffffffffa07b6062>] ?
>> gfs2_lm_withdraw+0x102/0x130 [gfs2]
>> Oct 24 13:02:06 host1 kernel: [<ffffffff814feb58>] ?
>> out_of_line_wait_on_bit+0x78/0x90
>> Oct 24 13:02:06 host1 kernel: [<ffffffff81092110>] ?
>> wake_bit_function+0x0/0x50
>> Oct 24 13:02:06 host1 kernel: [<ffffffffa07b60d0>] ?
>> gfs2_io_error_bh_i+0x40/0x50 [gfs2]
>> Oct 24 13:02:06 host1 kernel: [<ffffffff811ae0b6>] ?
>> __wait_on_buffer+0x26/0x30
>> Oct 24 13:02:06 host1 kernel: [<ffffffffa079e288>] ?
>> log_write_header+0x3a8/0x490 [gfs2]
>> Oct 24 13:02:06 host1 kernel: [<ffffffffa079e951>] ?
>> gfs2_log_flush+0x301/0x6f0 [gfs2]
>> Oct 24 13:02:06 host1 kernel: [<ffffffff811a48a9>] ?
>> sync_inodes_sb+0x149/0x190
>> Oct 24 13:02:06 host1 kernel: [<ffffffff810920d0>] ?
>> autoremove_wake_function+0x0/0x40
>> Oct 24 13:02:06 host1 kernel: [<ffffffffa07b24cc>] ?
> gfs2_sync_fs+0x2c/0x30
>> [gfs2]
>> Oct 24 13:02:06 host1 kernel: [<ffffffff811aa2e3>] ?
>> __sync_filesystem+0x53/0x90
>> Oct 24 13:02:06 host1 kernel: [<ffffffff811aa51b>] ?
>> sync_filesystem+0x4b/0x70
>> Oct 24 13:02:06 host1 kernel: [<ffffffff811b514e>] ? fsync_bdev+0x2e/0x70
>> Oct 24 13:02:06 host1 kernel: [<ffffffff8125ea9b>] ?
>> blkdev_ioctl+0x3ab/0x6e0
>> Oct 24 13:02:06 host1 kernel: [<ffffffff811b391c>] ? block_ioctl+0x3c/0x40
>> Oct 24 13:02:06 host1 kernel: [<ffffffff8118dff2>] ? vfs_ioctl+0x22/0xa0
>> Oct 24 13:02:06 host1 kernel: [<ffffffff8118e194>] ?
> do_vfs_ioctl+0x84/0x580
>> Oct 24 13:02:06 host1 kernel: [<ffffffff8118e711>] ? sys_ioctl+0x81/0xa0
>> Oct 24 13:02:06 host1 kernel: [<ffffffff8100b0f2>] ?
>> system_call_fastpath+0x16/0x1b
>> Oct 24 13:02:06 host1 abrt-dump-oops: Reported 4 kernel oopses to Abrt
>> Oct 24 13:02:29 host1 kernel: __ratelimit: 562 callbacks suppressed
>> Oct 24 13:02:29 host1 kernel: kvm: 23115: cpu0 unhandled wrmsr: 0x198 data
> 0
>> --- >8 ---
>>
>> Now I have this "uncontrolled entry" and it doesn't allow me to start cman
>> again:
>> --- 8< ---
>> Oct 24 18:07:42 host1 dlm_controld[16712]: dlm_controld 3.0.12.1 started
>> Oct 24 18:07:43 host1 gfs_controld[16761]: gfs_controld 3.0.12.1 started
>> Oct 24 18:07:43 host1 gfs_controld[16761]: found uncontrolled gfs2 fs
>> ckvm1_pod1:vg_psv4
>> Oct 24 18:07:43 host1 gfs_controld[16761]: telling cman to remove nodeid 1
>> from cluster
>> Oct 24 18:07:43 host1 corosync[16628]: cman killed by node 1 because we
> were
>> killed by cman_tool or other application
>> Oct 24 18:07:43 host1 dlm_controld[16712]: cluster is down, exiting
>> Oct 24 18:07:43 host1 dlm_controld[16712]: daemon cpg_dispatch error 2
>> --- >8 ---
>>
>> How to get rid of it? How to make node to "forget" about this gfs2?
>>
>> Thank you!
>>
>> -----Original Message-----
>> From: linux-cluster-bounces at redhat.com
>> [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Digimer
>> Sent: Thursday, October 24, 2013 5:20 PM
>> To: linux clustering
>> Subject: Re: [Linux-cluster] uncontrolled entry
>>
>> On 24/10/13 10:18, Vladimir Melnik wrote:
>>> Dear colleagues,
>>>
>>> I'll be very grateful for any clue or advice. I have a cluster
>>> (GFS2/DLM on iSCSI). Something happened today, so one node has stopped
>>> working in cluster and all operations with the filesystem was giving
>> "input/output errors".
>>> Other nodes are working okay.
>>>
>>> I unmounted the filesystem and stopped all services (gfs2, clvmd,
>>> cman, iscsi, iscsid), but I can't start cman again. That's what I have
>>> in
>>> log-files:
>>> fenced[2801]: found uncontrolled entry
>>> /sys/fs/gfs2/ckvm1_pod1:vg_psv4
>>> gfs_controld[2878]: found uncontrolled gfs2 fs ckvm1_pod1:vg_psv4
>>>
>>> Yes, I really have this "uncontrolled entry", how to get rid of it
>>> without rebooting the node?
>>>
>>> Thank you very much for any ideas and hints.
>>
>> I've not seen that particular error before. Can you paste your
> cluster.conf
>> and the log files from the effected node, starting just before the event
>> occured? Please also look at the logs from other
>> node(s) at the same time and paste how they responded to the node failure.
>>
>> --
>> Digimer
>> Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is
>> trapped in the mind of a person without access to education?
>>
>> --
>> Linux-cluster mailing list
>> Linux-cluster at redhat.com
>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>
>
>
--
Digimer
Papers and Projects: https://alteeve.ca/w/
What if the cure for cancer is trapped in the mind of a person without
access to education?
More information about the Linux-cluster
mailing list