[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [Linux-cluster] Freeze with cluster-2.03.11



On Fri, 27 Mar 2009, Bob Peterson wrote:

> Perhaps you should change your post_fail_delay to some very high
> number, recreate the problem, and when it freezes force a
> sysrq-trigger to get call traces for all the processes.
> Then also you can look at the dmesg to see if there was a kernel
> panic or something on the node that would otherwise be
> immediately fenced.

I enabled more kernel debugging, netconsole and captured the attaced 
console log. I hope it gives the required info.

Best regards,
Jozsef
--
E-mail : kadlec mail kfki hu, kadlec blackhole kfki hu
PGP key: http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address: KFKI Research Institute for Particle and Nuclear Physics
         H-1525 Budapest 114, POB. 49, Hungary
=============================================
[ INFO: possible recursive locking detected ]
2.6.27.21 #6
---------------------------------------------
dlm_controld/3536 is trying to acquire lock:
 (&sb->s_type->i_mutex_key#8/2){--..}, at: [<f891842b>] configfs_attach_group+0x43/0x18f [configfs]

but task is already holding lock:
 (&sb->s_type->i_mutex_key#8/2){--..}, at: [<f891842b>] configfs_attach_group+0x43/0x18f [configfs]

other info that might help us debug this:
2 locks held by dlm_controld/3536:
 #0:  (&sb->s_type->i_mutex_key#7/1){--..}, at: [<c0177f6e>] lookup_create+0x1b/0x85
 #1:  (&sb->s_type->i_mutex_key#8/2){--..}, at: [<f891842b>] configfs_attach_group+0x43/0x18f [configfs]

stack backtrace:
Pid: 3536, comm: dlm_controld Not tainted 2.6.27.21 #6
 [<c013e3a3>] validate_chain+0x9bd/0xf35
 [<c013eb96>] __lock_acquire+0x27b/0x932
 [<c013f2ac>] lock_acquire+0x5f/0x77
 [<f891842b>] configfs_attach_group+0x43/0x18f [configfs]
 [<c031857d>] mutex_lock_nested+0xa1/0x25c
 [<f891842b>] configfs_attach_group+0x43/0x18f [configfs]
 [<f891842b>] configfs_attach_group+0x43/0x18f [configfs]
 [<f891842b>] configfs_attach_group+0x43/0x18f [configfs]
 [<f89184cd>] configfs_attach_group+0xe5/0x18f [configfs]
 [<f8918ef2>] configfs_mkdir+0x1fb/0x36f [configfs]
 [<c01783d3>] vfs_mkdir+0x9c/0xdf
 [<c0319867>] _spin_unlock+0x14/0x1c
 [<c017a3a4>] sys_mkdirat+0xc5/0xd2
 [<c01f4a80>] trace_hardirqs_on_thunk+0xc/0x10
 [<c017a3d0>] sys_mkdir+0x1f/0x23
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
aoe: e1.0: setting 8704 byte data frames on eth1:0030486543a9
aoe: e2.0: setting 8704 byte data frames on eth1:003048654382
aoe: e2.0: setting 8704 byte data frames on eth1:003048654383
aoe: 0030486543a9 e1.0 vace0 has 2930382333 sectors
aoe: 003048654382 e2.0 vace0 has 2930382333 sectors
 etherd/e1.0: unknown partition table
 etherd/e2.0: unknown partition table
dlm: Using TCP for communications
dlm: got connection from 1
dlm: got connection from 5
dlm: connecting to 2
dlm: got connection from 4
dlm: got connection from 2
Trying to join cluster "lock_dlm", "kfki:home"
Joined cluster. Now mounting FS...
GFS: fsid=kfki:home.3: jid=3: Trying to acquire journal lock...
GFS: fsid=kfki:home.3: jid=3: Looking at journal...
GFS: fsid=kfki:home.3: jid=3: Done
Trying to join cluster "lock_dlm", "kfki:services"
Joined cluster. Now mounting FS...
GFS: fsid=kfki:services.3: jid=3: Trying to acquire journal lock...
GFS: fsid=kfki:services.3: jid=3: Looking at journal...
GFS: fsid=kfki:services.3: jid=3: Done
GFS: fsid=kfki:home.3: fast statfs start time = 1238196765
GFS: fsid=kfki:services.3: fast statfs start time = 1238196770
SysRq : Show State
  task                PC stack   pid father
init          S f7061b14     0     1      0
       f705f110 00000086 00000002 f7061b14 f7061b08 00000000 00000002 c045f0c0 
       c0461d00 c0461d00 c0461d00 f7061b18 f705f264 c373ed00 f6a8dd00 00000000 
       ffff9d54 00000000 00000246 c0318668 ffffffff 00000002 00000000 00000000 
Call Trace:
 [<c0318668>] mutex_lock_nested+0x18c/0x25c
 [<c0317dc6>] schedule_timeout+0x69/0xa1
 [<c0199808>] inotify_poll+0x45/0x4b
 [<c017cbae>] do_select+0x3ef/0x4cb
 [<c017c7d3>] do_select+0x14/0x4cb
 [<c017cfbf>] __pollwait+0x0/0xba
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c02c4724>] net_rx_action+0x85/0x1db
 [<c02c473c>] net_rx_action+0x9d/0x1db
 [<c0124797>] __do_softirq+0xa2/0xf9
 [<c01f3746>] number+0x25b/0x264
 [<c0103aa8>] restore_nocheck_notrace+0x0/0xe
 [<c014007b>] lockdep_stats_show+0x299/0x4f0
 [<c01300d8>] param_set_short+0x9/0x36
 [<c01f34ed>] number+0x2/0x264
 [<c01f4051>] vsnprintf+0x2dc/0x581
 [<c013dd7b>] validate_chain+0x395/0xf35
 [<c017ce7d>] core_sys_select+0x1f3/0x335
 [<c017ccac>] core_sys_select+0x22/0x335
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c0122b69>] do_wait+0x31e/0x342
 [<c017d0b8>] sys_select+0x3f/0x19e
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
kthreadd      S f7063fac     0     2      0
       f705e8a0 00000086 00000002 f7063fac f7063fa0 00000000 f705e8a0 c045f0c0 
       c0461d00 c0461d00 c0461d00 f7063fb0 f705e9f4 c373ed00 f7896d00 00000246 
       ffff989b 00000000 00000000 00000000 ffffffff 00000246 00000000 00000000 
Call Trace:
 [<c01314e3>] kthreadd+0x14a/0x14f
 [<c0131399>] kthreadd+0x0/0x14f
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
migration/0   R running      0     3      2
       f705e030 00000082 00000086 00000002 00000000 c3734d10 f705e030 c045f0c0 
       c0461d00 c0461d00 c0461d00 00000046 f705e184 c3734d00 f7967800 00000046 
       00000000 00000002 00000001 00000000 c011cefe 00000046 00000046 c3734d00 
Call Trace:
 [<c011cefe>] migration_thread+0x50/0x2cc
 [<c011cff5>] migration_thread+0x147/0x2cc
 [<c011ceae>] migration_thread+0x0/0x2cc
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
ksoftirqd/0   S f706bfa8     0     4      2
       f7069190 00000092 00000002 f706bfa8 f706bf9c 00000000 00000246 c045f0c0 
       c0461d00 c0461d00 c0461d00 f706bfac f70692e4 c3734d00 f7801800 c040caa0 
       ffffef69 00000000 c045f100 c0124797 ffffffff c0461b80 00000000 00000000 
Call Trace:
 [<c0124797>] __do_softirq+0xa2/0xf9
 [<c01249e5>] ksoftirqd+0xee/0xf3
 [<c01248f7>] ksoftirqd+0x0/0xf3
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
migration/1   S 00000002     0     5      2
       f7068920 00000082 00000086 00000002 00000000 c373ed10 f7068920 c045f0c0 
       c0461d00 c0461d00 c0461d00 00000046 f7068a74 c373ed00 f64a8580 00000046 
       00000000 00000002 00000001 00000000 c011cefe 00000046 00000046 c373ed00 
Call Trace:
 [<c011cefe>] migration_thread+0x50/0x2cc
 [<c011cff5>] migration_thread+0x147/0x2cc
 [<c011ceae>] migration_thread+0x0/0x2cc
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
ksoftirqd/1   S f7071fa8     0     6      2
       f70680b0 00000092 00000002 f7071fa8 f7071f9c 00000000 00000246 c045f0c0 
       c0461d00 c0461d00 c0461d00 f7071fac f7068204 c373ed00 f7c60800 c040caa0 
       ffffef7b 00000000 c045f100 c0124797 ffffffff c0461b80 00000000 00000000 
Call Trace:
 [<c0124797>] __do_softirq+0xa2/0xf9
 [<c01249e5>] ksoftirqd+0xee/0xf3
 [<c01248f7>] ksoftirqd+0x0/0xf3
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
events/0      R running      0     7      2
       f70729a0 00000082 00000002 f7079f94 f7079f88 00000000 00000046 c045f0c0 
       c0461d00 c0461d00 c0461d00 f7079f98 f7072af4 c3734d00 f78a5080 00000286 
       ffffef43 00000000 c0319cc1 00000000 ffffffff 00000046 00000000 00000000 
Call Trace:
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c012ee03>] worker_thread+0x0/0x89
 [<c012ee7d>] worker_thread+0x7a/0x89
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c012ee03>] worker_thread+0x0/0x89
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
events/1      R running      0     8      2
       f7072130 00000082 00013362 c373baa0 00000002 f707bf90 00000046 c045f0c0 
       c0461d00 c0461d00 c0461d00 f707bf98 f7072284 c373ed00 f7c60800 00000286 
       ffffef7e 00000000 c0319cc1 00000000 ffffffff 00000046 00000000 00000000 
Call Trace:
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c012ee03>] worker_thread+0x0/0x89
 [<c012ee7d>] worker_thread+0x7a/0x89
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c012ee03>] worker_thread+0x0/0x89
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
khelper       S 00000001     0     9      2
       f707d290 00000082 f707d610 00000001 00000002 00000001 00000046 c045f0c0 
       c0461d00 c0461d00 c0461d00 00000000 f707d3e4 c3734d00 f781a080 00000286 
       00000001 f702b39c c0319cc1 00000000 f707d290 c0319d8f c012ee03 c013d34e 
Call Trace:
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c0319d8f>] _spin_unlock_irqrestore+0x34/0x39
 [<c012ee03>] worker_thread+0x0/0x89
 [<c013d34e>] trace_hardirqs_on_caller+0x9d/0x113
 [<c012ee03>] worker_thread+0x0/0x89
 [<c012ee7d>] worker_thread+0x7a/0x89
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c012ee03>] worker_thread+0x0/0x89
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
kblockd/0     S 00000000     0    47      2
       f707ca20 00000082 00000046 00000000 00000002 00000001 00000046 c045f0c0 
       c0461d00 c0461d00 c0461d00 00000000 f707cb74 c3734d00 f7c60300 00000286 
       00000001 f70a651c c0319cc1 00000000 00000046 00000046 f70a6524 00000286 
Call Trace:
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c012ee03>] worker_thread+0x0/0x89
 [<c012ee7d>] worker_thread+0x7a/0x89
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c012ee03>] worker_thread+0x0/0x89
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
kblockd/1     S f70cbf94     0    48      2
       f707c1b0 00000082 00000002 f70cbf94 f70cbf88 00000000 00000046 c045f0c0 
       c0461d00 c0461d00 c0461d00 f70cbf98 f707c304 c373ed00 f781a300 00000286 
       ffff9376 00000000 c0319cc1 00000000 ffffffff c0319d8f 00000000 00000000 
Call Trace:
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c0319d8f>] _spin_unlock_irqrestore+0x34/0x39
 [<c012ee03>] worker_thread+0x0/0x89
 [<c012ee7d>] worker_thread+0x7a/0x89
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c012ee03>] worker_thread+0x0/0x89
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
kacpid        S f70cff94     0    49      2
       f7083390 00000082 00000002 f70cff94 f70cff88 00000000 00000046 c045f0c0 
       c0461d00 c0461d00 c0461d00 f70cff98 f70834e4 c373ed00 c03e81a0 00000286 
       ffff8af2 00000000 c0319cc1 00000000 ffffffff c0319d8f 00000000 00000000 
Call Trace:
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c0319d8f>] _spin_unlock_irqrestore+0x34/0x39
 [<c012ee03>] worker_thread+0x0/0x89
 [<c012ee7d>] worker_thread+0x7a/0x89
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c012ee03>] worker_thread+0x0/0x89
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
kacpi_notify  S f70d1f94     0    50      2
       f7082b20 00000082 00000002 f70d1f94 f70d1f88 00000000 00000046 c045f0c0 
       c0461d00 c0461d00 c0461d00 f70d1f98 f7082c74 c373ed00 c03e81a0 00000286 
       ffff8af2 00000000 c0319cc1 00000000 ffffffff c0319d8f 00000000 00000000 
Call Trace:
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c0319d8f>] _spin_unlock_irqrestore+0x34/0x39
 [<c012ee03>] worker_thread+0x0/0x89
 [<c012ee7d>] worker_thread+0x7a/0x89
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c012ee03>] worker_thread+0x0/0x89
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
ata/0         S 00000001     0   130      2
       f7120da0 00000082 f7121120 00000001 00000002 00000000 00000046 c045f0c0 
       c0461d00 c0461d00 c0461d00 00000000 f7120ef4 c3734d00 c03e81a0 00000286 
       00000001 f710dd1c c0319cc1 00000000 f7120da0 c0319d8f c012ee03 c013d34e 
Call Trace:
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c0319d8f>] _spin_unlock_irqrestore+0x34/0x39
 [<c012ee03>] worker_thread+0x0/0x89
 [<c013d34e>] trace_hardirqs_on_caller+0x9d/0x113
 [<c012ee03>] worker_thread+0x0/0x89
 [<c012ee7d>] worker_thread+0x7a/0x89
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c012ee03>] worker_thread+0x0/0x89
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
ata/1         S f7125f94     0   131      2
       f7121610 00000082 00000002 f7125f94 f7125f88 00000000 00000046 c045f0c0 
       c0461d00 c0461d00 c0461d00 f7125f98 f7121764 c373ed00 c03e81a0 00000286 
       ffff8af2 00000000 c0319cc1 00000000 ffffffff c0319d8f 00000000 00000000 
Call Trace:
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c0319d8f>] _spin_unlock_irqrestore+0x34/0x39
 [<c012ee03>] worker_thread+0x0/0x89
 [<c012ee7d>] worker_thread+0x7a/0x89
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c012ee03>] worker_thread+0x0/0x89
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
ata_aux       S f712df94     0   132      2
       f711a4b0 00000082 00000002 f712df94 f712df88 00000000 00000046 c045f0c0 
       c0461d00 c0461d00 c0461d00 f712df98 f711a604 c3734d00 c03e81a0 00000286 
       ffff8af2 00000000 c0319cc1 00000000 ffffffff c0319d8f 00000000 00000000 
Call Trace:
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c0319d8f>] _spin_unlock_irqrestore+0x34/0x39
 [<c012ee03>] worker_thread+0x0/0x89
 [<c012ee7d>] worker_thread+0x7a/0x89
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c012ee03>] worker_thread+0x0/0x89
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
ksuspend_usbd S 00000001     0   133      2
       f711ad20 00000082 f711b0a0 00000001 00000002 00000000 00000046 c045f0c0 
       c0461d00 c0461d00 c0461d00 00000000 f711ae74 c3734d00 c03e81a0 00000286 
       00000001 f710da1c c0319cc1 00000000 f711ad20 c0319d8f c012ee03 c013d34e 
Call Trace:
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c0319d8f>] _spin_unlock_irqrestore+0x34/0x39
 [<c012ee03>] worker_thread+0x0/0x89
 [<c013d34e>] trace_hardirqs_on_caller+0x9d/0x113
 [<c012ee03>] worker_thread+0x0/0x89
 [<c012ee7d>] worker_thread+0x7a/0x89
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c012ee03>] worker_thread+0x0/0x89
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
khubd         S c373baa0     0   139      2
       f711d690 00000082 00000e14 c373baa0 00000002 f7153f0c 00000046 c045f0c0 
       c0461d00 c0461d00 c0461d00 f7153f14 f711d7e4 c373ed00 f6b40300 00000286 
       ffff8ceb 00000000 c0319cc1 00000000 ffffffff c0319d8f 00000000 00000000 
Call Trace:
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c0319d8f>] _spin_unlock_irqrestore+0x34/0x39
 [<c027684d>] hub_thread+0x87f/0xcef
 [<c013d34e>] trace_hardirqs_on_caller+0x9d/0x113
 [<c0319c5a>] _spin_unlock_irq+0x20/0x23
 [<c011c3d4>] finish_task_switch+0x58/0xab
 [<c03177b2>] schedule+0x228/0x708
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c0275fce>] hub_thread+0x0/0xcef
 [<c013d34e>] trace_hardirqs_on_caller+0x9d/0x113
 [<c0275fce>] hub_thread+0x0/0xcef
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
kseriod       S c3731aa0     0   142      2
       f711ce20 00000082 00000400 c3731aa0 00000002 f7157f80 00000046 c045f0c0 
       c0461d00 c0461d00 c0461d00 f7157f88 f711cf74 c3734d00 f781a580 00000286 
       ffff8ca9 00000000 c0319cc1 00000000 ffffffff c0319d8f 00000000 00000000 
Call Trace:
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c0319d8f>] _spin_unlock_irqrestore+0x34/0x39
 [<c0284fde>] serio_thread+0xa4/0x2c6
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c0284f3a>] serio_thread+0x0/0x2c6
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
pdflush       S 00000002     0   181      2
       f7158920 00000082 00000092 00000002 00000000 c03edd8c f7158920 c045f0c0 
       c0461d00 c0461d00 c0461d00 00000046 f7158a74 c3734d00 c03e81a0 00000046 
       00000000 00000002 00000001 00000000 c0154020 f7158920 c0319c5a c0153fc2 
Call Trace:
 [<c0154020>] pdflush+0x5e/0x19c
 [<c0319c5a>] _spin_unlock_irq+0x20/0x23
 [<c0153fc2>] pdflush+0x0/0x19c
 [<c013d34e>] trace_hardirqs_on_caller+0x9d/0x113
 [<c0153fc2>] pdflush+0x0/0x19c
 [<c0154074>] pdflush+0xb2/0x19c
 [<c013d34e>] trace_hardirqs_on_caller+0x9d/0x113
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
pdflush       R running      0   182      2
       f7159190 00000082 00000002 f71c3f94 f71c3f88 00000000 f7159190 c045f0c0 
       c0461d00 c0461d00 c0461d00 f71c3f98 f71592e4 c3734d00 f7c60300 00000046 
       ffffee63 00000000 00000001 00000000 ffffffff 00000046 00000000 00000000 
Call Trace:
 [<c0153fc2>] pdflush+0x0/0x19c
 [<c0154074>] pdflush+0xb2/0x19c
 [<c0153b80>] wb_kupdate+0x0/0xe3
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
kswapd0       S f71e7f28     0   183      2
       f7130030 00000096 00000002 f71e7f28 f71e7f1c 00000000 00000046 c045f0c0 
       c0461d00 c0461d00 c0461d00 f71e7f2c f7130184 c373ed00 c03e81a0 00000286 
       ffff8af2 00000000 c0319cc1 00000000 ffffffff c0319d8f 00000000 00000000 
Call Trace:
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c0319d8f>] _spin_unlock_irqrestore+0x34/0x39
 [<c0157515>] kswapd+0x3fd/0x420
 [<c03177b2>] schedule+0x228/0x708
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c0319d8f>] _spin_unlock_irqrestore+0x34/0x39
 [<c0157118>] kswapd+0x0/0x420
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
aio/0         S 00000001     0   184      2
       f71308a0 00000082 f7130c20 00000001 00000002 00000000 00000046 c045f0c0 
       c0461d00 c0461d00 c0461d00 00000000 f71309f4 c3734d00 c03e81a0 00000286 
       00000001 f730dc1c c0319cc1 00000000 f71308a0 c0319d8f c012ee03 c013d34e 
Call Trace:
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c0319d8f>] _spin_unlock_irqrestore+0x34/0x39
 [<c012ee03>] worker_thread+0x0/0x89
 [<c013d34e>] trace_hardirqs_on_caller+0x9d/0x113
 [<c012ee03>] worker_thread+0x0/0x89
 [<c012ee7d>] worker_thread+0x7a/0x89
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c012ee03>] worker_thread+0x0/0x89
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
aio/1         S f732ff94     0   185      2
       f7131110 00000082 00000002 f732ff94 f732ff88 00000000 00000046 c045f0c0 
       c0461d00 c0461d00 c0461d00 f732ff98 f7131264 c373ed00 c03e81a0 00000286 
       ffff8af2 00000000 c0319cc1 00000000 ffffffff c0319d8f 00000000 00000000 
Call Trace:
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c0319d8f>] _spin_unlock_irqrestore+0x34/0x39
 [<c012ee03>] worker_thread+0x0/0x89
 [<c012ee7d>] worker_thread+0x7a/0x89
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c012ee03>] worker_thread+0x0/0x89
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
kpsmoused     S f6addf94     0   335      2
       f6aa8a20 00000082 00000002 f6addf94 f6addf88 00000000 00000046 c045f0c0 
       c0461d00 c0461d00 c0461d00 f6addf98 f6aa8b74 c373ed00 c03e81a0 00000286 
       ffff8bde 00000000 c0319cc1 00000000 ffffffff c0319d8f 00000000 00000000 
Call Trace:
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c0319d8f>] _spin_unlock_irqrestore+0x34/0x39
 [<c012ee03>] worker_thread+0x0/0x89
 [<c012ee7d>] worker_thread+0x7a/0x89
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c012ee03>] worker_thread+0x0/0x89
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
kstriped      S f6b03f94     0   339      2
       f6a780b0 00000082 00000002 f6b03f94 f6b03f88 00000000 00000046 c045f0c0 
       c0461d00 c0461d00 c0461d00 f6b03f98 f6a78204 c373ed00 c03e81a0 00000286 
       ffff8be4 00000000 c0319cc1 00000000 ffffffff c0319d8f 00000000 00000000 
Call Trace:
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c0319d8f>] _spin_unlock_irqrestore+0x34/0x39
 [<c012ee03>] worker_thread+0x0/0x89
 [<c012ee7d>] worker_thread+0x7a/0x89
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c012ee03>] worker_thread+0x0/0x89
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
edac-poller   S f6b0df94     0   343      2
       f6aa09a0 00000082 00000002 f6b0df94 f6b0df88 00000000 00000046 c045f0c0 
       c0461d00 c0461d00 c0461d00 f6b0df98 f6aa0af4 c3734d00 c03e81a0 00000286 
       ffff8be3 00000000 c0319cc1 00000000 ffffffff c0319d8f 00000000 00000000 
Call Trace:
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c0319d8f>] _spin_unlock_irqrestore+0x34/0x39
 [<c012ee03>] worker_thread+0x0/0x89
 [<c012ee7d>] worker_thread+0x7a/0x89
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c012ee03>] worker_thread+0x0/0x89
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
scsi_eh_0     S f79f7f68     0  1224      2
       f7999410 00000096 00000002 f79f7f68 f79f7f5c 00000000 00000000 c045f0c0 
       c0461d00 c0461d00 c0461d00 f79f7f6c f7999564 c3734d00 f7903a80 f780f840 
       ffff8d11 00000000 f780f800 00000000 ffffffff 00000286 00000000 00000000 
Call Trace:
 [<c0259a7b>] scsi_error_handler+0x0/0x2de
 [<c0259ac6>] scsi_error_handler+0x4b/0x2de
 [<c0117dce>] __wake_up_common+0x46/0x68
 [<c0319d8f>] _spin_unlock_irqrestore+0x34/0x39
 [<c0259a7b>] scsi_error_handler+0x0/0x2de
 [<c013d34e>] trace_hardirqs_on_caller+0x9d/0x113
 [<c0259a7b>] scsi_error_handler+0x0/0x2de
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
scsi_eh_1     S f7c4bf68     0  1225      2
       f7a0a330 00000096 00000002 f7c4bf68 f7c4bf5c 00000000 00000000 c045f0c0 
       c0461d00 c0461d00 c0461d00 f7c4bf6c f7a0a484 c3734d00 f78a5800 f7c53840 
       ffff8d7c 00000000 f7c53800 00000000 ffffffff 00000286 00000000 00000000 
Call Trace:
 [<c0259a7b>] scsi_error_handler+0x0/0x2de
 [<c0259ac6>] scsi_error_handler+0x4b/0x2de
 [<c0117dce>] __wake_up_common+0x46/0x68
 [<c0319d8f>] _spin_unlock_irqrestore+0x34/0x39
 [<c0259a7b>] scsi_error_handler+0x0/0x2de
 [<c013d34e>] trace_hardirqs_on_caller+0x9d/0x113
 [<c0259a7b>] scsi_error_handler+0x0/0x2de
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
scsi_eh_2     S f7c53000     0  1226      2
       f7a1b510 00000096 f7c53050 f7c53000 00000000 00000046 00000000 c045f0c0 
       c0461d00 c0461d00 c0461d00 f7a1b510 f7a1b664 c373ed00 f7982800 f7c53040 
       00000286 00000286 f7c53000 00000000 c025645c 00000286 00000282 f7c53000 
Call Trace:
 [<c025645c>] __scsi_iterate_devices+0x49/0x62
 [<c0259a7b>] scsi_error_handler+0x0/0x2de
 [<c0259a7b>] scsi_error_handler+0x0/0x2de
 [<c0259ac6>] scsi_error_handler+0x4b/0x2de
 [<c0117dce>] __wake_up_common+0x46/0x68
 [<c0319d8f>] _spin_unlock_irqrestore+0x34/0x39
 [<c0259a7b>] scsi_error_handler+0x0/0x2de
 [<c013d34e>] trace_hardirqs_on_caller+0x9d/0x113
 [<c0259a7b>] scsi_error_handler+0x0/0x2de
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
scsi_eh_3     S f78fd800     0  1227      2
       f6a79190 00000096 f78fd850 f78fd800 00000000 00000046 00000000 c045f0c0 
       c0461d00 c0461d00 c0461d00 f6a79190 f6a792e4 c373ed00 f6b40300 f78fd840 
       00000286 00000286 f78fd800 00000000 c025645c 00000286 00000282 f78fd800 
Call Trace:
 [<c025645c>] __scsi_iterate_devices+0x49/0x62
 [<c0259a7b>] scsi_error_handler+0x0/0x2de
 [<c0259a7b>] scsi_error_handler+0x0/0x2de
 [<c0259ac6>] scsi_error_handler+0x4b/0x2de
 [<c0117dce>] __wake_up_common+0x46/0x68
 [<c0319d8f>] _spin_unlock_irqrestore+0x34/0x39
 [<c0259a7b>] scsi_error_handler+0x0/0x2de
 [<c013d34e>] trace_hardirqs_on_caller+0x9d/0x113
 [<c0259a7b>] scsi_error_handler+0x0/0x2de
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
md2_raid1     S f7c6ff68     0  1391      2
       f7a96b20 00000096 00000002 f7c6ff68 f7c6ff5c 00000000 f29c2230 c045f0c0 
       c0461d00 c0461d00 c0461d00 f7c6ff6c f7a96c74 c3734d00 f7967080 c01f4716 
       ffffef43 00000000 f79d7a18 00000001 ffffffff 00000046 00000000 00000000 
Call Trace:
 [<c01f4716>] __delay+0x6/0x7
 [<c02a19c8>] md_thread+0x0/0xda
 [<c0317dc6>] schedule_timeout+0x69/0xa1
 [<c02a1a78>] md_thread+0xb0/0xda
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c02a19c8>] md_thread+0x0/0xda
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
md2_resync    R running      0  1392      2
       f7a74130 00000086 00000046 00000000 00000002 00000001 c04b9e00 c045f0c0 
       c0461d00 c0461d00 c0461d00 00000000 f7a74284 c3734d00 f781a300 c04b9e00 
       00000286 c04b9e00 00000286 00000286 f7d57e28 c04b9e00 c0128a5a 00000000 
Call Trace:
 [<c0128a5a>] __mod_timer+0x94/0xa3
 [<c0317da1>] schedule_timeout+0x44/0xa1
 [<c01f4771>] delay_tsc+0x2a/0x47
 [<c01287ec>] process_timeout+0x0/0x5
 [<c0128aa3>] msleep_interruptible+0x22/0x2f
 [<c02979af>] sync_request+0x5c7/0x795
 [<c029abd8>] is_mddev_idle+0x103/0x10f
 [<c029aade>] is_mddev_idle+0x9/0x10f
 [<c02973e8>] sync_request+0x0/0x795
 [<c02a1dc2>] md_do_sync+0x320/0xdb7
 [<c013dd7b>] validate_chain+0x395/0xf35
 [<c013eb96>] __lock_acquire+0x27b/0x932
 [<c0319c5a>] _spin_unlock_irq+0x20/0x23
 [<c02a19c8>] md_thread+0x0/0xda
 [<c013d34e>] trace_hardirqs_on_caller+0x9d/0x113
 [<c02a19c8>] md_thread+0x0/0xda
 [<c02a19ea>] md_thread+0x22/0xda
 [<c0319d8f>] _spin_unlock_irqrestore+0x34/0x39
 [<c02a19c8>] md_thread+0x0/0xda
 [<c013d34e>] trace_hardirqs_on_caller+0x9d/0x113
 [<c02a19c8>] md_thread+0x0/0xda
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
md1_raid1     S f7445f68     0  1401      2
       f79b1510 00000096 00000002 f7445f68 f7445f5c 00000000 f7073210 c045f0c0 
       c0461d00 c0461d00 c0461d00 f7445f6c f79b1664 c373ed00 f64d7d00 c01f4716 
       ffffd981 00000000 f7463418 00000001 ffffffff 00000046 00000000 00000000 
Call Trace:
 [<c01f4716>] __delay+0x6/0x7
 [<c02a19c8>] md_thread+0x0/0xda
 [<c0317dc6>] schedule_timeout+0x69/0xa1
 [<c02a1a78>] md_thread+0xb0/0xda
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c02a19c8>] md_thread+0x0/0xda
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
md0_raid1     S f7957f68     0  1409      2
       f79f21b0 00000096 00000002 f7957f68 f7957f5c 00000000 f7073210 c045f0c0 
       c0461d00 c0461d00 c0461d00 f7957f6c f79f2304 c373ed00 f64d7d00 c01f4716 
       ffffd981 00000000 f7490d98 00000001 ffffffff 00000046 00000000 00000000 
Call Trace:
 [<c01f4716>] __delay+0x6/0x7
 [<c02a19c8>] md_thread+0x0/0xda
 [<c0317dc6>] schedule_timeout+0x69/0xa1
 [<c02a1a78>] md_thread+0xb0/0xda
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c02a19c8>] md_thread+0x0/0xda
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
kdmflush      S 00000001     0  1420      2
       f79d0630 00000082 f79d09b0 00000001 00000002 00000001 00000046 c045f0c0 
       c0461d00 c0461d00 c0461d00 00000000 f79d0784 c373ed00 f7801d00 00000286 
       00000001 f788629c c0319cc1 00000000 f79d0630 c0319d8f c012ee03 c013d34e 
Call Trace:
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c0319d8f>] _spin_unlock_irqrestore+0x34/0x39
 [<c012ee03>] worker_thread+0x0/0x89
 [<c013d34e>] trace_hardirqs_on_caller+0x9d/0x113
 [<c012ee03>] worker_thread+0x0/0x89
 [<c012ee7d>] worker_thread+0x7a/0x89
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c012ee03>] worker_thread+0x0/0x89
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
kdmflush      S 00000001     0  1426      2
       f71d0aa0 00000082 f71d0e20 00000001 00000002 00000001 00000046 c045f0c0 
       c0461d00 c0461d00 c0461d00 00000000 f71d0bf4 c373ed00 f7801d00 00000286 
       00000001 f7c5e51c c0319cc1 00000000 f71d0aa0 c0319d8f c012ee03 c013d34e 
Call Trace:
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c0319d8f>] _spin_unlock_irqrestore+0x34/0x39
 [<c012ee03>] worker_thread+0x0/0x89
 [<c013d34e>] trace_hardirqs_on_caller+0x9d/0x113
 [<c012ee03>] worker_thread+0x0/0x89
 [<c012ee7d>] worker_thread+0x7a/0x89
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c012ee03>] worker_thread+0x0/0x89
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
kdmflush      S 00000001     0  1432      2
       f7a75210 00000082 f7a75590 00000001 00000002 00000001 00000046 c045f0c0 
       c0461d00 c0461d00 c0461d00 00000000 f7a75364 c373ed00 f6b40a80 00000286 
       00000001 f74cf79c c0319cc1 00000000 f7a75210 c0319d8f c012ee03 c013d34e 
Call Trace:
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c0319d8f>] _spin_unlock_irqrestore+0x34/0x39
 [<c012ee03>] worker_thread+0x0/0x89
 [<c013d34e>] trace_hardirqs_on_caller+0x9d/0x113
 [<c012ee03>] worker_thread+0x0/0x89
 [<c012ee7d>] worker_thread+0x7a/0x89
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c012ee03>] worker_thread+0x0/0x89
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
kjournald     R running      0  1514      2
       f7a264b0 00000082 00000046 f78c84d8 00000001 f78c8510 00000046 c045f0c0 
       c0461d00 c0461d00 c0461d00 00000000 f7a26604 c3734d00 f7967080 00000286 
       00000001 f78c8510 c0319cc1 00000000 00000046 00000246 00000246 f78c8414 
Call Trace:
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c01cae8a>] kjournald+0x1bc/0x1c5
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c01cacce>] kjournald+0x0/0x1c5
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
udevd         S f759db14     0  1675      1
       f7a43110 00000086 00000002 f759db14 f759db08 00000000 00000000 c045f0c0 
       c0461d00 c0461d00 c0461d00 f759db18 f7a43264 c373ed00 f7896d00 f78d8800 
       ffff989b 00000000 00000000 00000002 ffffffff f78d8800 00000000 00000000 
Call Trace:
 [<c0317dc6>] schedule_timeout+0x69/0xa1
 [<c0176343>] pipe_poll+0x24/0x86
 [<c017cbae>] do_select+0x3ef/0x4cb
 [<c017c7d3>] do_select+0x14/0x4cb
 [<c017cfbf>] __pollwait+0x0/0xba
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c01f4051>] vsnprintf+0x2dc/0x581
 [<c01f3746>] number+0x25b/0x264
 [<c014cc6d>] add_to_page_cache_locked+0x46/0x8f
 [<c01f4051>] vsnprintf+0x2dc/0x581
 [<c031995f>] _spin_lock+0x31/0x3c
 [<c015bec9>] handle_mm_fault+0x4ea/0x8a4
 [<c017ce7d>] core_sys_select+0x1f3/0x335
 [<c017ccac>] core_sys_select+0x22/0x335
 [<c0319867>] _spin_unlock+0x14/0x1c
 [<c015a727>] do_wp_page+0x2a2/0x537
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c0122a32>] do_wait+0x1e7/0x342
 [<c011377d>] do_page_fault+0xa5/0x848
 [<c017d0b8>] sys_select+0x3f/0x19e
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
gfs2_scand    R running      0  2420      2
       f7a83290 00000092 00000046 00000000 00000002 00000001 f706e000 c045f0c0 
       c0461d00 c0461d00 c0461d00 00000000 f7a833e4 c373ed00 f64a8a80 f706e000 
       00000282 f706e000 00000282 00000282 f75cbfac f706e000 c0128a5a 00000000 
Call Trace:
 [<c0128a5a>] __mod_timer+0x94/0xa3
 [<f8da03c8>] gfs2_scand+0x0/0x63 [gfs2]
 [<c0317da1>] schedule_timeout+0x44/0xa1
 [<c0319826>] _read_unlock+0x14/0x1c
 [<c01287ec>] process_timeout+0x0/0x5
 [<f8da041e>] gfs2_scand+0x56/0x63 [gfs2]
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
glock_workque S c3731aa0     0  2422      2
       f7a82a20 00000082 00000afb c3731aa0 00000002 f7585f90 00000046 c045f0c0 
       c0461d00 c0461d00 c0461d00 f7585f98 f7a82b74 c3734d00 f64d7d00 00000286 
       ffff9049 00000000 c0319cc1 00000000 ffffffff c0319d8f 00000000 00000000 
Call Trace:
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c0319d8f>] _spin_unlock_irqrestore+0x34/0x39
 [<c012ee03>] worker_thread+0x0/0x89
 [<c012ee7d>] worker_thread+0x7a/0x89
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c012ee03>] worker_thread+0x0/0x89
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
glock_workque S 00000001     0  2423      2
       f7120530 00000082 f71208b0 00000001 00000002 00000000 00000046 c045f0c0 
       c0461d00 c0461d00 c0461d00 00000000 f7120684 c373ed00 f78ff080 00000286 
       00000001 f501be9c c0319cc1 00000000 f7120530 c0319d8f c012ee03 c013d34e 
Call Trace:
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c0319d8f>] _spin_unlock_irqrestore+0x34/0x39
 [<c012ee03>] worker_thread+0x0/0x89
 [<c013d34e>] trace_hardirqs_on_caller+0x9d/0x113
 [<c012ee03>] worker_thread+0x0/0x89
 [<c012ee7d>] worker_thread+0x7a/0x89
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c012ee03>] worker_thread+0x0/0x89
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
ksnapd        S 00000001     0  2536      2
       f79db110 00000082 f79db490 00000001 00000002 00000000 00000046 c045f0c0 
       c0461d00 c0461d00 c0461d00 00000000 f79db264 c373ed00 f7963a80 00000286 
       00000001 f501b89c c0319cc1 00000000 f79db110 c0319d8f c012ee03 c013d34e 
Call Trace:
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c0319d8f>] _spin_unlock_irqrestore+0x34/0x39
 [<c012ee03>] worker_thread+0x0/0x89
 [<c013d34e>] trace_hardirqs_on_caller+0x9d/0x113
 [<c012ee03>] worker_thread+0x0/0x89
 [<c012ee7d>] worker_thread+0x7a/0x89
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c012ee03>] worker_thread+0x0/0x89
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
kjournald     S f6533f84     0  2623      2
       f6a48ca0 00000082 00000002 f6533f84 f6533f78 00000000 00000046 c045f0c0 
       c0461d00 c0461d00 c0461d00 f6533f88 f6a48df4 c3734d00 f64a8a80 f6a48ca0 
       ffff9450 00000000 f5065d10 c013d220 ffffffff 00000246 00000000 00000000 
Call Trace:
 [<c013d220>] mark_held_locks+0x62/0x73
 [<c01cae8a>] kjournald+0x1bc/0x1c5
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c01cacce>] kjournald+0x0/0x1c5
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
kjournald     S f50b7f84     0  2624      2
       f6a70030 00000082 00000002 f50b7f84 f50b7f78 00000000 00000046 c045f0c0 
       c0461d00 c0461d00 c0461d00 f50b7f88 f6a70184 c3734d00 f7c60a80 00000286 
       ffffeba1 00000000 c0319cc1 00000000 ffffffff 00000246 00000000 00000000 
Call Trace:
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c01cae8a>] kjournald+0x1bc/0x1c5
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c01cacce>] kjournald+0x0/0x1c5
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
kjournald     R running      0  2625      2
       f71e1410 00000082 00000046 f506bcd8 00000001 f506bd10 00000046 c045f0c0 
       c0461d00 c0461d00 c0461d00 00000000 f71e1564 c3734d00 f7967080 00000286 
       00000001 f506bd10 c0319cc1 00000000 00000046 00000246 00000246 f506bc14 
Call Trace:
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c01cae8a>] kjournald+0x1bc/0x1c5
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c01cacce>] kjournald+0x0/0x1c5
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
getty         S f50e3eb4     0  2962      1
       f6ab8b20 00000082 00000002 f50e3eb4 f50e3ea8 00000000 00000000 c045f0c0 
       c0461d00 c0461d00 c0461d00 f50e3eb8 f6ab8c74 c373ed00 f78a5300 00000000 
       ffff9238 00000000 00000046 f78e797c ffffffff f7538000 00000000 00000000 
Call Trace:
 [<c0317dc6>] schedule_timeout+0x69/0xa1
 [<c013d220>] mark_held_locks+0x62/0x73
 [<c0319d8f>] _spin_unlock_irqrestore+0x34/0x39
 [<c013d34e>] trace_hardirqs_on_caller+0x9d/0x113
 [<c022ff41>] read_chan+0x1e1/0x640
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c022c03a>] tty_read+0x70/0x9d
 [<c022fd60>] read_chan+0x0/0x640
 [<c022bfca>] tty_read+0x0/0x9d
 [<c01715f8>] vfs_read+0x85/0x11b
 [<c017192b>] sys_read+0x41/0x6a
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
getty         S 0038a000     0  2963      1
       f7a6b190 00000082 00000001 0038a000 00000000 c013eb96 00000000 c045f0c0 
       c0461d00 c0461d00 c0461d00 00000000 f7a6b2e4 c373ed00 f7982800 00000000 
       f7a6b190 c03182b9 00000046 f65b897c f65b8800 f744d000 00000046 00000000 
Call Trace:
 [<c013eb96>] __lock_acquire+0x27b/0x932
 [<c03182b9>] mutex_lock_interruptible_nested+0x192/0x2b7
 [<c0317dc6>] schedule_timeout+0x69/0xa1
 [<c013d220>] mark_held_locks+0x62/0x73
 [<c0319d8f>] _spin_unlock_irqrestore+0x34/0x39
 [<c013d34e>] trace_hardirqs_on_caller+0x9d/0x113
 [<c022ff41>] read_chan+0x1e1/0x640
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c022c03a>] tty_read+0x70/0x9d
 [<c022fd60>] read_chan+0x0/0x640
 [<c022bfca>] tty_read+0x0/0x9d
 [<c01715f8>] vfs_read+0x85/0x11b
 [<c017192b>] sys_read+0x41/0x6a
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
login         S 00000002     0  2965      1
       f7a2eda0 00000082 f7a2eda0 00000002 00000000 c040c990 f7a2eda0 c045f0c0 
       c0461d00 c0461d00 c0461d00 00000246 f7a2eef4 c3734d00 f64d7d00 00000000 
       00000002 00000002 00000000 00000000 c01228e1 00000246 00000246 c040c980 
Call Trace:
 [<c01228e1>] do_wait+0x96/0x342
 [<c0122ad7>] do_wait+0x28c/0x342
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c0122bf6>] sys_wait4+0x69/0xa4
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
getty         S 0038a000     0  2966      1
       f7a26d20 00000082 00000001 0038a000 00000000 c013eb96 00000000 c045f0c0 
       c0461d00 c0461d00 c0461d00 00000000 f7a26e74 c3734d00 f7967d00 00000000 
       f7a26d20 c03182b9 00000046 f748897c f7488800 f6461000 00000046 00000000 
Call Trace:
 [<c013eb96>] __lock_acquire+0x27b/0x932
 [<c03182b9>] mutex_lock_interruptible_nested+0x192/0x2b7
 [<c0317dc6>] schedule_timeout+0x69/0xa1
 [<c013d220>] mark_held_locks+0x62/0x73
 [<c0319d8f>] _spin_unlock_irqrestore+0x34/0x39
 [<c013d34e>] trace_hardirqs_on_caller+0x9d/0x113
 [<c022ff41>] read_chan+0x1e1/0x640
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c022c03a>] tty_read+0x70/0x9d
 [<c022fd60>] read_chan+0x0/0x640
 [<c022bfca>] tty_read+0x0/0x9d
 [<c01715f8>] vfs_read+0x85/0x11b
 [<c017192b>] sys_read+0x41/0x6a
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
login         S 00000002     0  2969      1
       f7a02b20 00000082 f7a02b20 00000002 00000000 c040c990 f7a02b20 c045f0c0 
       c0461d00 c0461d00 c0461d00 00000246 f7a02c74 c3734d00 f7801300 00000000 
       00000002 00000002 00000000 00000000 c01228e1 00000246 00000246 c040c980 
Call Trace:
 [<c01228e1>] do_wait+0x96/0x342
 [<c0122ad7>] do_wait+0x28c/0x342
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c0122bf6>] sys_wait4+0x69/0xa4
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
getty         S 0038a000     0  2970      1
       f78e45b0 00000082 00000001 0038a000 00000000 c013eb96 00000000 c045f0c0 
       c0461d00 c0461d00 c0461d00 00000000 f78e4704 c3734d00 f6505d00 00000000 
       f78e45b0 c03182b9 00000046 f7c0b17c f7c0b000 f6460000 00000046 00000000 
Call Trace:
 [<c013eb96>] __lock_acquire+0x27b/0x932
 [<c03182b9>] mutex_lock_interruptible_nested+0x192/0x2b7
 [<c0317dc6>] schedule_timeout+0x69/0xa1
 [<c013d220>] mark_held_locks+0x62/0x73
 [<c0319d8f>] _spin_unlock_irqrestore+0x34/0x39
 [<c013d34e>] trace_hardirqs_on_caller+0x9d/0x113
 [<c022ff41>] read_chan+0x1e1/0x640
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c022c03a>] tty_read+0x70/0x9d
 [<c022fd60>] read_chan+0x0/0x640
 [<c022bfca>] tty_read+0x0/0x9d
 [<c01715f8>] vfs_read+0x85/0x11b
 [<c017192b>] sys_read+0x41/0x6a
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
syslogd       R running      0  3016      1
       f71d8b20 00000086 f75be31c f71d8b20 00000002 00000000 f6d175ac c045f0c0 
       c0461d00 c0461d00 c0461d00 00000020 f71d8c74 c3734d00 f7896300 00000000 
       00000000 c030b41c 00000246 00000246 f75be30c f75be30c 00000000 00000020 
Call Trace:
 [<c030b41c>] unix_peer_get+0x11/0x2b
 [<c0319867>] _spin_unlock+0x14/0x1c
 [<c0317dc6>] schedule_timeout+0x69/0xa1
 [<c017cbae>] do_select+0x3ef/0x4cb
 [<c017c7d3>] do_select+0x14/0x4cb
 [<c017cfbf>] __pollwait+0x0/0xba
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c01b5d3f>] __ext3_get_inode_loc+0xba/0x307
 [<c01c2f12>] __ext3_journal_dirty_metadata+0x16/0x3b
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c01c5eab>] journal_stop+0x134/0x1c3
 [<c01b81e3>] ext3_ordered_write_end+0xf0/0x14a
 [<c014d45b>] generic_file_buffered_write+0x1a6/0x677
 [<c0185f44>] mnt_drop_write+0x1d/0xd8
 [<c031995f>] _spin_lock+0x31/0x3c
 [<c0176d05>] pipe_write+0x5c/0x450
 [<c017ce7d>] core_sys_select+0x1f3/0x335
 [<c017ccac>] core_sys_select+0x22/0x335
 [<c0176ca9>] pipe_write+0x0/0x450
 [<c0170bfa>] do_sync_readv_writev+0xcb/0x107
 [<c0185904>] mntput_no_expire+0x18/0xef
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c01f4de3>] copy_from_user+0x2d/0x59
 [<c0170a70>] rw_copy_check_uvector+0x64/0xf4
 [<c017d0b8>] sys_select+0x3f/0x19e
 [<c01290c3>] sigprocmask+0x68/0xc3
 [<c012b117>] sys_rt_sigprocmask+0xbf/0xd4
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
dd            S f6489f04     0  3049      1
       f6a605b0 00000086 00000002 f6489f04 f6489ef8 00000000 00000046 c045f0c0 
       c0461d00 c0461d00 c0461d00 f6489f08 f6a60704 c373ed00 f78ff580 00000282 
       ffff93bc 00000000 c0319cc1 00000000 ffffffff c0319d8f 00000000 00000000 
Call Trace:
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c0319d8f>] _spin_unlock_irqrestore+0x34/0x39
 [<c01216f2>] do_syslog+0x278/0x399
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c01ae2f4>] kmsg_read+0x0/0x36
 [<c01a62de>] proc_reg_read+0x58/0x79
 [<c01a6286>] proc_reg_read+0x0/0x79
 [<c01715f8>] vfs_read+0x85/0x11b
 [<c017192b>] sys_read+0x41/0x6a
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
klogd         S 00000000     0  3050      1
       f6aa9290 00000092 f6aa9628 00000000 f6aa9610 00000001 00000046 c045f0c0 
       c0461d00 c0461d00 c0461d00 00000000 f6aa93e4 c373ed00 f7967580 f6aa9290 
       00000000 00000000 f6aa9290 c0318467 f75e8fd8 c013d34e f75e8fd4 f75e8ff4 
Call Trace:
 [<c0318467>] __mutex_unlock_slowpath+0x89/0xf9
 [<c013d34e>] trace_hardirqs_on_caller+0x9d/0x113
 [<c0176c87>] pipe_wait+0x50/0x72
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c0177205>] pipe_read+0xe3/0x3fe
 [<c0170e16>] do_sync_read+0xd2/0x10e
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c01a4b6f>] dnotify_parent+0x1f/0x5f
 [<c0170d44>] do_sync_read+0x0/0x10e
 [<c01715f8>] vfs_read+0x85/0x11b
 [<c017192b>] sys_read+0x41/0x6a
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
mysqld_safe   S f5dcff24     0  3150      1
       f6ab9390 00000082 00000002 f5dcff24 f5dcff18 00000000 f6ab9390 c045f0c0 
       c0461d00 c0461d00 c0461d00 f5dcff28 f6ab94e4 c3734d00 f7832a80 00000000 
       ffff9273 00000000 00000000 00000000 ffffffff 00000246 00000000 00000000 
Call Trace:
 [<c0122ad7>] do_wait+0x28c/0x342
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c0122bf6>] sys_wait4+0x69/0xa4
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
mysqld        S f50dfb14     0  3192   3150
       f6aa1210 00000086 00000002 f50dfb14 00000002 00000000 f540132c c045f0c0 
       c0461d00 c0461d00 c0461d00 f50dfb18 f6aa1364 c3734d00 f781a300 00000000 
       00000046 00000000 00000002 00000001 f540131c 00000282 f7854280 00000000 
Call Trace:
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c0317dc6>] schedule_timeout+0x69/0xa1
 [<c030cb2e>] unix_poll+0x1a/0xab
 [<c017cbae>] do_select+0x3ef/0x4cb
 [<c017c7d3>] do_select+0x14/0x4cb
 [<c017cfbf>] __pollwait+0x0/0xba
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c018ff76>] __getblk+0x36/0x2a7
 [<c013dd7b>] validate_chain+0x395/0xf35
 [<c013eb96>] __lock_acquire+0x27b/0x932
 [<c01b6ea8>] ext3_getblk+0xb9/0x1c5
 [<c0190aa9>] ll_rw_block+0x42/0x10e
 [<c02b9896>] release_sock+0x24/0xac
 [<c03199a0>] _spin_lock_bh+0x36/0x41
 [<c02e0dfa>] ip_setsockopt+0x113/0xbad
 [<c01248bb>] local_bh_enable_ip+0x79/0xb5
 [<c02e0dfa>] ip_setsockopt+0x113/0xbad
 [<c017ce7d>] core_sys_select+0x1f3/0x335
 [<c017ccac>] core_sys_select+0x22/0x335
 [<c011a997>] enqueue_task_fair+0xa2/0xb4
 [<c011e008>] wake_up_new_task+0x87/0xa0
 [<c011f996>] do_fork+0xce/0x247
 [<c017d0b8>] sys_select+0x3f/0x19e
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
mysqld        S f5de7df8     0  3200   3150
       f7a42030 00000096 00000002 f5de7df8 f5de7dec 00000000 f7a42030 c045f0c0 
       c0461d00 c0461d00 c0461d00 f5de7dfc f7a42184 c373ed00 f781a300 f5de7e3c 
       ffff9298 00000000 08e3d9f8 c0319cc1 ffffffff c0319d8f 00000000 00000000 
Call Trace:
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c0319d8f>] _spin_unlock_irqrestore+0x34/0x39
 [<c0140c19>] futex_wait+0x318/0x396
 [<c013eb96>] __lock_acquire+0x27b/0x932
 [<c013eb96>] __lock_acquire+0x27b/0x932
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c0141b9a>] do_futex+0xbc/0x951
 [<c011a997>] enqueue_task_fair+0xa2/0xb4
 [<c0319d8f>] _spin_unlock_irqrestore+0x34/0x39
 [<c013d34e>] trace_hardirqs_on_caller+0x9d/0x113
 [<c01f2957>] rwsem_wake+0x4d/0x116
 [<c03196ce>] call_rwsem_wake+0xa/0xc
 [<c0134edc>] up_read+0x26/0x2a
 [<c01424b5>] sys_futex+0x86/0x111
 [<c01f4a80>] trace_hardirqs_on_thunk+0xc/0x10
 [<c013d34e>] trace_hardirqs_on_caller+0x9d/0x113
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
mysqld        S 00000002     0  3201   3150
       f7a27590 00000096 f7a27590 00000002 00000000 c082d374 f7a27590 c045f0c0 
       c0461d00 c0461d00 c0461d00 00000046 f7a276e4 c3734d00 f781a300 f5081e3c 
       00000282 00000000 08e3da68 c0319cc1 00000046 00000046 f5081e3c 00000282 
Call Trace:
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c0140c19>] futex_wait+0x318/0x396
 [<c011abae>] try_to_wake_up+0xa8/0x137
 [<c0319867>] _spin_unlock+0x14/0x1c
 [<c0140e83>] futex_requeue+0xdb/0x251
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c0141b9a>] do_futex+0xbc/0x951
 [<c014c6d9>] find_get_pages_tag+0x11/0x140
 [<c01b84be>] ext3_write_inode+0x0/0x34
 [<c0152aae>] mapping_tagged+0x58/0x60
 [<c0154861>] pagevec_lookup_tag+0x24/0x2c
 [<c014d0df>] wait_on_page_writeback_range+0x5f/0x104
 [<c0319867>] _spin_unlock+0x14/0x1c
 [<c018c009>] sync_inode+0x25/0x2a
 [<c01424b5>] sys_futex+0x86/0x111
 [<c018ef5e>] do_fsync+0x73/0x92
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
mysqld        S f79e8d20     0  3202   3150
       f79e89a0 00000096 f79e89a0 f79e8d20 00000001 c082c6e8 f79e89a0 c045f0c0 
       c0461d00 c0461d00 c0461d00 00000046 f79e8af4 c3734d00 f781a300 f75dde3c 
       00000282 00000000 08e3dad8 c0319cc1 f79e89a0 c0319d8f 00000000 c013d34e 
Call Trace:
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c0319d8f>] _spin_unlock_irqrestore+0x34/0x39
 [<c013d34e>] trace_hardirqs_on_caller+0x9d/0x113
 [<c0140c19>] futex_wait+0x318/0x396
 [<c0152137>] __alloc_pages_internal+0xb6/0x41f
 [<c013dd7b>] validate_chain+0x395/0xf35
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c0141b9a>] do_futex+0xbc/0x951
 [<c013eb96>] __lock_acquire+0x27b/0x932
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c01a4b6f>] dnotify_parent+0x1f/0x5f
 [<c01424b5>] sys_futex+0x86/0x111
 [<c0171656>] vfs_read+0xe3/0x11b
 [<c01f4a80>] trace_hardirqs_on_thunk+0xc/0x10
 [<c013d34e>] trace_hardirqs_on_caller+0x9d/0x113
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
mysqld        S 00000002     0  3203   3150
       f7ac7610 00000096 f7ac7610 00000002 00000000 c082cea4 f7ac7610 c045f0c0 
       c0461d00 c0461d00 c0461d00 00000046 f7ac7764 c3734d00 f781a300 f64e3e3c 
       00000282 00000000 08e3db48 c0319cc1 00000046 00000046 f64e3e3c 00000282 
Call Trace:
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c0140c19>] futex_wait+0x318/0x396
 [<c014e2e5>] generic_file_aio_write+0x55/0xd8
 [<c014e303>] generic_file_aio_write+0x73/0xd8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c0141b9a>] do_futex+0xbc/0x951
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c01a4b6f>] dnotify_parent+0x1f/0x5f
 [<c01424b5>] sys_futex+0x86/0x111
 [<c0171540>] vfs_write+0xe5/0x118
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
mysqld        R running      0  3205   3150
       f79e9210 00000086 00000002 f7985b14 f7985b08 00000000 c04b9e00 c045f0c0 
       c0461d00 c0461d00 c0461d00 f7985b18 f79e9364 c3734d00 f781a300 c04b9e00 
       ffffef43 00000000 00000286 00000286 ffffffff c04b9e00 00000000 00000000 
Call Trace:
 [<c0317da1>] schedule_timeout+0x44/0xa1
 [<c01287ec>] process_timeout+0x0/0x5
 [<c017cbae>] do_select+0x3ef/0x4cb
 [<c017c7d3>] do_select+0x14/0x4cb
 [<c013eb96>] __lock_acquire+0x27b/0x932
 [<c013dd7b>] validate_chain+0x395/0xf35
 [<c013d220>] mark_held_locks+0x62/0x73
 [<c0151ee7>] get_page_from_freelist+0x304/0x48d
 [<c013d34e>] trace_hardirqs_on_caller+0x9d/0x113
 [<c017cfbf>] __pollwait+0x0/0xba
 [<c013eb96>] __lock_acquire+0x27b/0x932
 [<c011a997>] enqueue_task_fair+0xa2/0xb4
 [<c011abae>] try_to_wake_up+0xa8/0x137
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c0128c10>] del_timer+0x4b/0x53
 [<c02598c4>] scsi_delete_timer+0xb/0x1b
 [<c012ed34>] queue_work_on+0x34/0x43
 [<c011a997>] enqueue_task_fair+0xa2/0xb4
 [<c017ce7d>] core_sys_select+0x1f3/0x335
 [<c017ccac>] core_sys_select+0x22/0x335
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c025a9b9>] scsi_next_command+0x25/0x2f
 [<c025ab90>] scsi_end_request+0x52/0x75
 [<c025b6dc>] scsi_io_completion+0x87/0x3de
 [<c031995f>] _spin_lock+0x31/0x3c
 [<c017d14f>] sys_select+0xd6/0x19e
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
mysqld        R running      0  3206   3150
       f7a1a430 00000086 00000002 f6583b14 f6583b08 00000000 c04b9e00 c045f0c0 
       c0461d00 c0461d00 c0461d00 f6583b18 f7a1a584 c3734d00 f781a300 c04b9e00 
       ffffeed6 00000000 00000286 00000286 ffffffff c04b9e00 00000000 00000000 
Call Trace:
 [<c0317da1>] schedule_timeout+0x44/0xa1
 [<c01287ec>] process_timeout+0x0/0x5
 [<c017cbae>] do_select+0x3ef/0x4cb
 [<c017c7d3>] do_select+0x14/0x4cb
 [<c017cfbf>] __pollwait+0x0/0xba
 [<c011abae>] try_to_wake_up+0xa8/0x137
 [<c0131642>] autoremove_wake_function+0x14/0x35
 [<c0117dce>] __wake_up_common+0x46/0x68
 [<c012ed34>] queue_work_on+0x34/0x43
 [<c02edfe6>] tcp_rcv_established+0x341/0x70b
 [<c02f3582>] tcp_v4_do_rcv+0x94/0x1af
 [<c0319867>] _spin_unlock+0x14/0x1c
 [<c02f4f5a>] tcp_v4_rcv+0x498/0x5be
 [<c02db3f7>] ip_local_deliver_finish+0x10d/0x15f
 [<c02db329>] ip_local_deliver_finish+0x3f/0x15f
 [<c0108bbd>] nommu_map_single+0x0/0x63
 [<c017ce7d>] core_sys_select+0x1f3/0x335
 [<c017ccac>] core_sys_select+0x22/0x335
 [<f89483e7>] e1000_clean_rx_irq+0x301/0x41a [e1000]
 [<f89480e6>] e1000_clean_rx_irq+0x0/0x41a [e1000]
 [<f89459a9>] e1000_clean+0x450/0x5f9 [e1000]
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c025ab90>] scsi_end_request+0x52/0x75
 [<c02c4724>] net_rx_action+0x85/0x1db
 [<c02c473c>] net_rx_action+0x9d/0x1db
 [<c017d14f>] sys_select+0xd6/0x19e
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
mysqld        S 00000002     0  3207   3150
       f6a61690 00000096 f6a61690 00000002 00000000 c082d348 f6a61690 c045f0c0 
       c0461d00 c0461d00 c0461d00 00000046 f6a617e4 c3734d00 f781a300 f50d7e3c 
       00000282 00000000 08af80b0 c0319cc1 00000046 00000046 f50d7e3c 00000282 
Call Trace:
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c0140c19>] futex_wait+0x318/0x396
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c0140d19>] futex_wake+0x52/0xe1
 [<c0319867>] _spin_unlock+0x14/0x1c
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c0141b9a>] do_futex+0xbc/0x951
 [<c025a9b9>] scsi_next_command+0x25/0x2f
 [<c025b6dc>] scsi_io_completion+0x87/0x3de
 [<c031995f>] _spin_lock+0x31/0x3c
 [<c01424b5>] sys_futex+0x86/0x111
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
mysqld        S f5117ecc     0  3208   3150
       f79f2a20 00000082 00000002 f5117ecc f5117ec0 00000000 f7073210 c045f0c0 
       c0461d00 c0461d00 c0461d00 f5117ed0 f79f2b74 c373ed00 f781a300 00000000 
       ffffec7b 00000000 c373bf30 f79f2a20 ffffffff 00000000 00000000 00000000 
Call Trace:
 [<c0317dc6>] schedule_timeout+0x69/0xa1
 [<c012a940>] dequeue_signal+0x3c/0x12a
 [<c012b2be>] sys_rt_sigtimedwait+0x87/0x285
 [<c012b37c>] sys_rt_sigtimedwait+0x145/0x285
 [<c012907a>] sigprocmask+0x1f/0xc3
 [<c0319c5a>] _spin_unlock_irq+0x20/0x23
 [<c01290c3>] sigprocmask+0x68/0xc3
 [<c012b117>] sys_rt_sigprocmask+0xbf/0xd4
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
mysqld        S f651fd94     0  3209   3150
       f79a43b0 00000082 00000002 f651fd94 f651fd88 00000000 00000000 c045f0c0 
       c0461d00 c0461d00 c0461d00 f651fd98 f79a4504 c3734d00 f781a300 00000002 
       ffffb5a6 00000000 f5dcd100 f5dcd12c ffffffff 00000000 00000000 00000000 
Call Trace:
 [<c0317dc6>] schedule_timeout+0x69/0xa1
 [<c02ba1f7>] sk_wait_data+0x54/0xa1
 [<c02ba20c>] sk_wait_data+0x69/0xa1
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c02e6654>] tcp_recvmsg+0x42c/0x7ff
 [<c02b926a>] sock_common_recvmsg+0x3e/0x54
 [<c02b7020>] sock_aio_read+0xdb/0xef
 [<c0170e16>] do_sync_read+0xd2/0x10e
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c0171687>] vfs_read+0x114/0x11b
 [<c017192b>] sys_read+0x41/0x6a
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
mysqld        S 00000002     0  3210   3150
       f79b84b0 00000096 f79b84b0 00000002 00000000 c082da80 f79b84b0 c045f0c0 
       c0461d00 c0461d00 c0461d00 00000046 f79b8604 c3734d00 f781a300 f5017e3c 
       00000282 00000000 08edccf4 c0319cc1 00000046 00000046 f5017e3c 00000282 
Call Trace:
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c0140c19>] futex_wait+0x318/0x396
 [<c014ddda>] __generic_file_aio_write_nolock+0x285/0x4df
 [<c014e2e5>] generic_file_aio_write+0x55/0xd8
 [<c014e303>] generic_file_aio_write+0x73/0xd8
 [<c0140d19>] futex_wake+0x52/0xe1
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c0141b9a>] do_futex+0xbc/0x951
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c01a4b6f>] dnotify_parent+0x1f/0x5f
 [<c01424b5>] sys_futex+0x86/0x111
 [<c01719bc>] sys_write+0x68/0x6a
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
mysqld        S 00000002     0  4053   3150
       f6a49510 00000096 f6a49510 00000002 00000000 c082c6bc f6a49510 c045f0c0 
       c0461d00 c0461d00 c0461d00 00000046 f6a49664 c3734d00 f781a300 f209fe3c 
       00000282 00000000 0875bba0 c0319cc1 00000046 00000046 f209fe3c 00000282 
Call Trace:
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c0140c19>] futex_wait+0x318/0x396
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c0141b9a>] do_futex+0xbc/0x951
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c01424b5>] sys_futex+0x86/0x111
 [<c0171952>] sys_read+0x68/0x6a
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
logger        S 00000000     0  3193   3150
       f6ab0230 00000092 f6ab05c8 00000000 f6ab05b0 00000001 00000046 c045f0c0 
       c0461d00 c0461d00 c0461d00 00000000 f6ab0384 c3734d00 f7896580 f6ab0230 
       00000000 00000000 f6ab0230 c0318467 f6ddd768 c013d34e f6ddd764 f6ddd784 
Call Trace:
 [<c0318467>] __mutex_unlock_slowpath+0x89/0xf9
 [<c013d34e>] trace_hardirqs_on_caller+0x9d/0x113
 [<c0176c87>] pipe_wait+0x50/0x72
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c0177205>] pipe_read+0xe3/0x3fe
 [<c013eb96>] __lock_acquire+0x27b/0x932
 [<c0170e16>] do_sync_read+0xd2/0x10e
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c011c3a2>] finish_task_switch+0x26/0xab
 [<c03177b2>] schedule+0x228/0x708
 [<c0170d44>] do_sync_read+0x0/0x10e
 [<c01715f8>] vfs_read+0x85/0x11b
 [<c017192b>] sys_read+0x41/0x6a
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
master        R running      0  3362      1
       f6aa0130 00000086 00000002 f5173f1c f5173f10 00000000 c04b9e00 c045f0c0 
       c0461d00 c0461d00 c0461d00 f5173f20 f6aa0284 c3734d00 f6a8d800 c04b9e00 
       ffffef68 00000000 00000282 00000282 ffffffff c04b9e00 00000000 00000000 
Call Trace:
 [<c0317da1>] schedule_timeout+0x44/0xa1
 [<c01287ec>] process_timeout+0x0/0x5
 [<c0199fe9>] sys_epoll_wait+0x15b/0x500
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
mdadm         R running      0  3403      1
       f79fd310 00000086 00000046 00000000 00000002 00000001 c04b9e00 c045f0c0 
       c0461d00 c0461d00 c0461d00 00000000 f79fd464 c3734d00 f7801d00 c04b9e00 
       00000286 c04b9e00 00000286 00000286 f7499b38 c04b9e00 c0128a5a 00000000 
Call Trace:
 [<c0128a5a>] __mod_timer+0x94/0xa3
 [<c0317da1>] schedule_timeout+0x44/0xa1
 [<c01287ec>] process_timeout+0x0/0x5
 [<c017cbae>] do_select+0x3ef/0x4cb
 [<c017c7d3>] do_select+0x14/0x4cb
 [<c017cfbf>] __pollwait+0x0/0xba
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c013dd7b>] validate_chain+0x395/0xf35
 [<c013dd7b>] validate_chain+0x395/0xf35
 [<c013d220>] mark_held_locks+0x62/0x73
 [<c0151ee7>] get_page_from_freelist+0x304/0x48d
 [<c013d34e>] trace_hardirqs_on_caller+0x9d/0x113
 [<c013dd7b>] validate_chain+0x395/0xf35
 [<c013dd7b>] validate_chain+0x395/0xf35
 [<c011a997>] enqueue_task_fair+0xa2/0xb4
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c02994f8>] md_wakeup_thread+0x25/0x29
 [<c02a0a52>] md_ioctl+0x10c/0x1082
 [<c0180f3a>] __d_lookup+0xbf/0x14e
 [<c0180f64>] __d_lookup+0xe9/0x14e
 [<c017ce7d>] core_sys_select+0x1f3/0x335
 [<c017ccac>] core_sys_select+0x22/0x335
 [<c0195268>] do_open+0xc5/0x2b7
 [<c01955ff>] blkdev_open+0x0/0x53
 [<c0195624>] blkdev_open+0x25/0x53
 [<c016f8f7>] __dentry_open+0x176/0x20a
 [<c016f9c0>] nameidata_to_filp+0x35/0x3f
 [<c0318668>] mutex_lock_nested+0x18c/0x25c
 [<c0171bb2>] file_kill+0x13/0x2d
 [<c017d14f>] sys_select+0xd6/0x19e
 [<c016f61c>] filp_close+0x3e/0x62
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
ccsd          S f6509b14     0  3461      1
       f79fcaa0 00000086 00000002 f6509b14 00000002 00000000 f55ef0ac c045f0c0 
       c0461d00 c0461d00 c0461d00 00000000 f79fcbf4 c373ed00 f756c300 00000000 
       00000046 00000000 00000002 00000001 f55ef09c 00000282 f7910280 00000000 
Call Trace:
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c0317dc6>] schedule_timeout+0x69/0xa1
 [<c030cb2e>] unix_poll+0x1a/0xab
 [<c017cbae>] do_select+0x3ef/0x4cb
 [<c017c7d3>] do_select+0x14/0x4cb
 [<c017cfbf>] __pollwait+0x0/0xba
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c013dd7b>] validate_chain+0x395/0xf35
 [<c013dd7b>] validate_chain+0x395/0xf35
 [<c013eb96>] __lock_acquire+0x27b/0x932
 [<c0136043>] sched_clock_cpu+0x10f/0x158
 [<c011826e>] update_curr+0x40/0x52
 [<c011abae>] try_to_wake_up+0xa8/0x137
 [<c0131642>] autoremove_wake_function+0x14/0x35
 [<c017ce7d>] core_sys_select+0x1f3/0x335
 [<c017ccac>] core_sys_select+0x22/0x335
 [<c0319c5a>] _spin_unlock_irq+0x20/0x23
 [<c011c3d4>] finish_task_switch+0x58/0xab
 [<c011c3a2>] finish_task_switch+0x26/0xab
 [<c03177b2>] schedule+0x228/0x708
 [<c017d0b8>] sys_select+0x3f/0x19e
 [<c016f61c>] filp_close+0x3e/0x62
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
ccsd          S f5105b14     0  3462      1
       f79d0ea0 00000086 00000002 f5105b14 f5105b08 00000000 f55f7aac c045f0c0 
       c0461d00 c0461d00 c0461d00 f5105b18 f79d0ff4 c3734d00 f756c300 00000000 
       ffff9486 00000000 00000002 00000001 ffffffff 00000282 00000000 00000000 
Call Trace:
 [<c0317dc6>] schedule_timeout+0x69/0xa1
 [<c030cb2e>] unix_poll+0x1a/0xab
 [<c017cbae>] do_select+0x3ef/0x4cb
 [<c017c7d3>] do_select+0x14/0x4cb
 [<c013eb96>] __lock_acquire+0x27b/0x932
 [<c017cfbf>] __pollwait+0x0/0xba
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c013dd7b>] validate_chain+0x395/0xf35
 [<c013c416>] check_noncircular+0xd/0xe7
 [<c013dd7b>] validate_chain+0x395/0xf35
 [<c013dd7b>] validate_chain+0x395/0xf35
 [<c013dd7b>] validate_chain+0x395/0xf35
 [<c013eb96>] __lock_acquire+0x27b/0x932
 [<c0136043>] sched_clock_cpu+0x10f/0x158
 [<c011826e>] update_curr+0x40/0x52
 [<c011abae>] try_to_wake_up+0xa8/0x137
 [<c0117dce>] __wake_up_common+0x46/0x68
 [<c013eb96>] __lock_acquire+0x27b/0x932
 [<c017ce7d>] core_sys_select+0x1f3/0x335
 [<c017ccac>] core_sys_select+0x22/0x335
 [<c013dd7b>] validate_chain+0x395/0xf35
 [<c02b7020>] sock_aio_read+0xdb/0xef
 [<c013dd7b>] validate_chain+0x395/0xf35
 [<c02b740e>] sockfd_lookup_light+0x24/0x41
 [<c013eb96>] __lock_acquire+0x27b/0x932
 [<c0319c5a>] _spin_unlock_irq+0x20/0x23
 [<c013d34e>] trace_hardirqs_on_caller+0x9d/0x113
 [<c0319c5a>] _spin_unlock_irq+0x20/0x23
 [<c011c3d4>] finish_task_switch+0x58/0xab
 [<c011c3a2>] finish_task_switch+0x26/0xab
 [<c03177b2>] schedule+0x228/0x708
 [<c01a4b6f>] dnotify_parent+0x1f/0x5f
 [<c017d0b8>] sys_select+0x3f/0x19e
 [<c01f4a80>] trace_hardirqs_on_thunk+0xc/0x10
 [<c013d34e>] trace_hardirqs_on_caller+0x9d/0x113
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
ntpd          S 00000000     0  3475      1
       f711c5b0 00200082 00200046 00000000 00000002 00000001 c04b9e00 c045f0c0 
       c0461d00 c0461d00 c0461d00 00000000 f711c704 c3734d00 f78ffa80 c04b9e00 
       00200282 c04b9e00 00200282 00200282 f5025c18 c04b9e00 c0128a5a 00000000 
Call Trace:
 [<c0128a5a>] __mod_timer+0x94/0xa3
 [<c0317da1>] schedule_timeout+0x44/0xa1
 [<c01287ec>] process_timeout+0x0/0x5
 [<c017c62b>] do_sys_poll+0x238/0x311
 [<c017cfbf>] __pollwait+0x0/0xba
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c02b9896>] release_sock+0x24/0xac
 [<c03199a0>] _spin_lock_bh+0x36/0x41
 [<c02b9896>] release_sock+0x24/0xac
 [<c03199a0>] _spin_lock_bh+0x36/0x41
 [<f8e8bdf3>] udpv6_recvmsg+0x1ac/0x335 [ipv6]
 [<c01248bb>] local_bh_enable_ip+0x79/0xb5
 [<f8e8bdf3>] udpv6_recvmsg+0x1ac/0x335 [ipv6]
 [<c011a997>] enqueue_task_fair+0xa2/0xb4
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c025a9b9>] scsi_next_command+0x25/0x2f
 [<c025ab90>] scsi_end_request+0x52/0x75
 [<c025b6dc>] scsi_io_completion+0x87/0x3de
 [<c031995f>] _spin_lock+0x31/0x3c
 [<c01a4b6f>] dnotify_parent+0x1f/0x5f
 [<c0171540>] vfs_write+0xe5/0x118
 [<c017c731>] sys_poll+0x2d/0x71
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
ntpd          S 00000000     0  3476      1
       f6a708a0 00200082 00200046 00000000 00000002 00000001 c04b9e00 c045f0c0 
       c0461d00 c0461d00 c0461d00 00000000 f6a709f4 c3734d00 f64d7800 c04b9e00 
       00200282 c04b9e00 00200282 00200282 f7457c18 c04b9e00 c0128a5a 00000000 
Call Trace:
 [<c0128a5a>] __mod_timer+0x94/0xa3
 [<c0317da1>] schedule_timeout+0x44/0xa1
 [<c01287ec>] process_timeout+0x0/0x5
 [<c017c62b>] do_sys_poll+0x238/0x311
 [<c017cfbf>] __pollwait+0x0/0xba
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c015a727>] do_wp_page+0x2a2/0x537
 [<c015bb92>] handle_mm_fault+0x1b3/0x8a4
 [<c031995f>] _spin_lock+0x31/0x3c
 [<c015bec9>] handle_mm_fault+0x4ea/0x8a4
 [<c011871b>] task_rq_lock+0x2e/0x55
 [<c031995f>] _spin_lock+0x31/0x3c
 [<c011abae>] try_to_wake_up+0xa8/0x137
 [<c0117dce>] __wake_up_common+0x46/0x68
 [<c0319826>] _read_unlock+0x14/0x1c
 [<c030d03b>] unix_stream_sendmsg+0x1c3/0x32b
 [<c02b6f28>] sock_aio_write+0xcb/0xe8
 [<c014cb52>] find_lock_page+0x25/0x55
 [<c02b6e5d>] sock_aio_write+0x0/0xe8
 [<c0170bfa>] do_sync_readv_writev+0xcb/0x107
 [<c015a227>] __do_fault+0x192/0x3f0
 [<c01a4b6f>] dnotify_parent+0x1f/0x5f
 [<c0171361>] do_readv_writev+0x118/0x179
 [<c02b6e5d>] sock_aio_write+0x0/0xe8
 [<c0134eca>] up_read+0x14/0x2a
 [<c011382e>] do_page_fault+0x156/0x848
 [<c01713fe>] vfs_writev+0x3c/0x50
 [<c017c731>] sys_poll+0x2d/0x71
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
sshd          S 00000000     0  3492      1
       f79da030 00200086 00000042 00000000 00000042 00000002 00000000 c045f0c0 
       c0461d00 c0461d00 c0461d00 00000032 f79da184 c373ed00 f756c580 f791b980 
       00000020 00200046 00000000 00000002 00000001 f5afd59c 00200282 f791b980 
Call Trace:
 [<c0317dc6>] schedule_timeout+0x69/0xa1
 [<c02e4b5d>] tcp_poll+0x17/0x11e
 [<c017cbae>] do_select+0x3ef/0x4cb
 [<c017c7d3>] do_select+0x14/0x4cb
 [<f8e87ecd>] ndisc_recv_ns+0x226/0x563 [ipv6]
 [<c017cfbf>] __pollwait+0x0/0xba
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c013eb96>] __lock_acquire+0x27b/0x932
 [<c013dd7b>] validate_chain+0x395/0xf35
 [<c013dd7b>] validate_chain+0x395/0xf35
 [<c013dd7b>] validate_chain+0x395/0xf35
 [<c013dd7b>] validate_chain+0x395/0xf35
 [<c013dd7b>] validate_chain+0x395/0xf35
 [<c013d220>] mark_held_locks+0x62/0x73
 [<c0151ee7>] get_page_from_freelist+0x304/0x48d
 [<c013d34e>] trace_hardirqs_on_caller+0x9d/0x113
 [<c013eb96>] __lock_acquire+0x27b/0x932
 [<c013d220>] mark_held_locks+0x62/0x73
 [<c0319c5a>] _spin_unlock_irq+0x20/0x23
 [<c013d34e>] trace_hardirqs_on_caller+0x9d/0x113
 [<c0319c5a>] _spin_unlock_irq+0x20/0x23
 [<c014cc97>] add_to_page_cache_locked+0x70/0x8f
 [<c01175e5>] kmap_atomic+0x1c/0x21
 [<c016ba36>] shmem_getpage+0x4e9/0x661
 [<c01175e5>] kmap_atomic+0x1c/0x21
 [<c013eb96>] __lock_acquire+0x27b/0x932
 [<c013eb96>] __lock_acquire+0x27b/0x932
 [<c017ce7d>] core_sys_select+0x1f3/0x335
 [<c017ccac>] core_sys_select+0x22/0x335
 [<c015a351>] __do_fault+0x2bc/0x3f0
 [<c015a227>] __do_fault+0x192/0x3f0
 [<c013d34e>] trace_hardirqs_on_caller+0x9d/0x113
 [<c015bb55>] handle_mm_fault+0x176/0x8a4
 [<c011377d>] do_page_fault+0xa5/0x848
 [<c017d0b8>] sys_select+0x3f/0x19e
 [<c01f4a80>] trace_hardirqs_on_thunk+0xc/0x10
 [<c01136d8>] do_page_fault+0x0/0x848
 [<c013d34e>] trace_hardirqs_on_caller+0x9d/0x113
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
pickup        S f6581f1c     0  3503   3362
       f79e40b0 00200086 00000002 f6581f1c f6581f10 00000000 c04b9e00 c045f0c0 
       c0461d00 c0461d00 c0461d00 f6581f20 f79e4204 c3734d00 f78ffd00 c04b9e00 
       ffffd9c4 00000000 00200282 00200282 ffffffff c04b9e00 00000000 00000000 
Call Trace:
 [<c0317da1>] schedule_timeout+0x44/0xa1
 [<c01287ec>] process_timeout+0x0/0x5
 [<c0199fe9>] sys_epoll_wait+0x15b/0x500
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
qmgr          R running      0  3504   3362
       f71d1310 00200086 00000002 f513ff1c f513ff10 00000000 f706e000 c045f0c0 
       c0461d00 c0461d00 c0461d00 f513ff20 f71d1464 c373ed00 f781a580 f706e000 
       ffffef69 00000000 00200282 00200282 ffffffff f706e000 00000000 00000000 
Call Trace:
 [<c0317da1>] schedule_timeout+0x44/0xa1
 [<c01287ec>] process_timeout+0x0/0x5
 [<c0199fe9>] sys_epoll_wait+0x15b/0x500
 [<c017bbcf>] vfs_ioctl+0x1f/0x6d
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
aisexec       R running      0  3510      1
       f79a4c20 00000082 00000400 c373baa0 00000002 f645fbf0 f706e000 c045f0c0 
       c0461d00 c0461d00 c0461d00 f645fbf8 f79a4d74 c373ed00 f64a8a80 f706e000 
       ffffef78 00000000 00000282 00000282 ffffffff f706e000 00000000 00000000 
Call Trace:
 [<c0317da1>] schedule_timeout+0x44/0xa1
 [<c01287ec>] process_timeout+0x0/0x5
 [<c017c62b>] do_sys_poll+0x238/0x311
 [<c017cfbf>] __pollwait+0x0/0xba
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c0108c32>] read_tsc+0x6/0x28
 [<c0136bb3>] getnstimeofday+0x44/0x100
 [<c01f5002>] copy_to_user+0x2f/0x4d
 [<c017c731>] sys_poll+0x2d/0x71
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
aisexec       S f750fbf4     0  3511      1
       f7aa4c20 00000082 00000002 f750fbf4 f750fbe8 00000000 f7aa4c20 c045f0c0 
       c0461d00 c0461d00 c0461d00 f750fbf8 f7aa4d74 c373ed00 f64a8a80 00000080 
       ffff93a4 00000000 00000002 c0555fa0 ffffffff 00000000 00000000 00000000 
Call Trace:
 [<c0317dc6>] schedule_timeout+0x69/0xa1
 [<c017c62b>] do_sys_poll+0x238/0x311
 [<c017cfbf>] __pollwait+0x0/0xba
 [<c013d220>] mark_held_locks+0x62/0x73
 [<c0151ee7>] get_page_from_freelist+0x304/0x48d
 [<c013d34e>] trace_hardirqs_on_caller+0x9d/0x113
 [<c013eb96>] __lock_acquire+0x27b/0x932
 [<c013dd7b>] validate_chain+0x395/0xf35
 [<c013d220>] mark_held_locks+0x62/0x73
 [<c013dd7b>] validate_chain+0x395/0xf35
 [<c013eb96>] __lock_acquire+0x27b/0x932
 [<c013c4fc>] find_usage_backwards+0xc/0xce
 [<c013c4fc>] find_usage_backwards+0xc/0xce
 [<c013c4fc>] find_usage_backwards+0xc/0xce
 [<c013eb96>] __lock_acquire+0x27b/0x932
 [<c013dd7b>] validate_chain+0x395/0xf35
 [<c013eb96>] __lock_acquire+0x27b/0x932
 [<c015a351>] __do_fault+0x2bc/0x3f0
 [<c015a227>] __do_fault+0x192/0x3f0
 [<c015bb55>] handle_mm_fault+0x176/0x8a4
 [<c011377d>] do_page_fault+0xa5/0x848
 [<c0134eca>] up_read+0x14/0x2a
 [<c011382e>] do_page_fault+0x156/0x848
 [<c01f4a80>] trace_hardirqs_on_thunk+0xc/0x10
 [<c01136d8>] do_page_fault+0x0/0x848
 [<c017c731>] sys_poll+0x2d/0x71
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
aisexec       S f4835df8     0  3513      1
       f6a51590 00000096 00000002 f4835df8 f4835dec 00000000 f6a51590 c045f0c0 
       c0461d00 c0461d00 c0461d00 f4835dfc f6a516e4 c373ed00 f64a8a80 f4835e3c 
       ffff9468 00000000 093b44c0 c0319cc1 ffffffff c0319d8f 00000000 00000000 
Call Trace:
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c0319d8f>] _spin_unlock_irqrestore+0x34/0x39
 [<c0140c19>] futex_wait+0x318/0x396
 [<c014ddda>] __generic_file_aio_write_nolock+0x285/0x4df
 [<c014e303>] generic_file_aio_write+0x73/0xd8
 [<c0140d19>] futex_wake+0x52/0xe1
 [<c013dd7b>] validate_chain+0x395/0xf35
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c0141b9a>] do_futex+0xbc/0x951
 [<c013eb96>] __lock_acquire+0x27b/0x932
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c01a4b6f>] dnotify_parent+0x1f/0x5f
 [<c01424b5>] sys_futex+0x86/0x111
 [<c01f4a80>] trace_hardirqs_on_thunk+0xc/0x10
 [<c013d34e>] trace_hardirqs_on_caller+0x9d/0x113
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
aisexec       S f510c998     0  3529      1
       f79b9590 00000086 00000246 f510c998 00000000 f510e000 00000246 c045f0c0 
       c0461d00 c0461d00 c0461d00 c01d720f f79b96e4 c373ed00 f64a8a80 00000000 
       f510c988 00000000 00000000 f510e000 c01d720f 00000246 00000246 00000246 
Call Trace:
 [<c01d720f>] ipc_lock+0x60/0xb8
 [<c01d720f>] ipc_lock+0x60/0xb8
 [<c01d933f>] sys_semtimedop+0x5a8/0x7a4
 [<c03177b2>] schedule+0x228/0x708
 [<c01409b7>] futex_wait+0xb6/0x396
 [<c0319867>] _spin_unlock+0x14/0x1c
 [<c01409ec>] futex_wait+0xeb/0x396
 [<c0140d19>] futex_wake+0x52/0xe1
 [<c0319867>] _spin_unlock+0x14/0x1c
 [<c0140d87>] futex_wake+0xc0/0xe1
 [<c0141b58>] do_futex+0x7a/0x951
 [<c013eb96>] __lock_acquire+0x27b/0x932
 [<c0107401>] sys_ipc+0x6a/0x293
 [<c01f4a80>] trace_hardirqs_on_thunk+0xc/0x10
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
aisexec       S f510c898     0  3541      1
       f6a71110 00000086 00000246 f510c898 00000000 f5106000 00000246 c045f0c0 
       c0461d00 c0461d00 c0461d00 c01d720f f6a71264 c3734d00 f64a8a80 00000000 
       f510c888 00000001 00000000 f5106000 c01d720f 00000246 00000246 00000246 
Call Trace:
 [<c01d720f>] ipc_lock+0x60/0xb8
 [<c01d720f>] ipc_lock+0x60/0xb8
 [<c01d933f>] sys_semtimedop+0x5a8/0x7a4
 [<c01f4de3>] copy_from_user+0x2d/0x59
 [<c01f4de3>] copy_from_user+0x2d/0x59
 [<c02bede0>] verify_iovec+0x2a/0x8a
 [<c02b7d4d>] sys_sendmsg+0x232/0x237
 [<c02b8c9e>] sys_socketcall+0xb7/0x2a9
 [<c0107401>] sys_ipc+0x6a/0x293
 [<c01f4a80>] trace_hardirqs_on_thunk+0xc/0x10
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
aisexec       S f5027d7c     0  3542      1
       f7998330 00000086 00000002 f5027d7c f5027d70 00000000 00000246 c045f0c0 
       c0461d00 c0461d00 c0461d00 f5027d80 f7998484 c373ed00 f64a8a80 00000000 
       ffff985c 00000000 00000000 f5026000 ffffffff 00000246 00000000 00000000 
Call Trace:
 [<c01d933f>] sys_semtimedop+0x5a8/0x7a4
 [<c0107401>] sys_ipc+0x6a/0x293
 [<c01f4a80>] trace_hardirqs_on_thunk+0xc/0x10
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
aisexec       S f510c798     0  3556      1
       f7a03390 00000086 00000246 f510c798 00000000 f48ac000 00000246 c045f0c0 
       c0461d00 c0461d00 c0461d00 c01d720f f7a034e4 c373ed00 f64a8a80 00000000 
       f510c788 00000003 00000000 f48ac000 c01d720f 00000246 00000246 00000246 
Call Trace:
 [<c01d720f>] ipc_lock+0x60/0xb8
 [<c01d720f>] ipc_lock+0x60/0xb8
 [<c01d933f>] sys_semtimedop+0x5a8/0x7a4
 [<c01f4de3>] copy_from_user+0x2d/0x59
 [<c01f4de3>] copy_from_user+0x2d/0x59
 [<c02bede0>] verify_iovec+0x2a/0x8a
 [<c02b7d4d>] sys_sendmsg+0x232/0x237
 [<c02b8c9e>] sys_socketcall+0xb7/0x2a9
 [<c0107401>] sys_ipc+0x6a/0x293
 [<c01f4a80>] trace_hardirqs_on_thunk+0xc/0x10
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
aisexec       S f4884a98     0  3653      1
       f6a60e20 00000086 00000246 f4884a98 00000000 f48d6000 00000246 c045f0c0 
       c0461d00 c0461d00 c0461d00 c01d720f f6a60f74 c3734d00 f64a8a80 00000000 
       f4884a88 00000004 00000000 f48d6000 c01d720f 00000246 00000246 00000246 
Call Trace:
 [<c01d720f>] ipc_lock+0x60/0xb8
 [<c01d720f>] ipc_lock+0x60/0xb8
 [<c01d933f>] sys_semtimedop+0x5a8/0x7a4
 [<c01f4de3>] copy_from_user+0x2d/0x59
 [<c01f4de3>] copy_from_user+0x2d/0x59
 [<c02bede0>] verify_iovec+0x2a/0x8a
 [<c02b7d4d>] sys_sendmsg+0x232/0x237
 [<c02b8c9e>] sys_socketcall+0xb7/0x2a9
 [<c0107401>] sys_ipc+0x6a/0x293
 [<c01f4a80>] trace_hardirqs_on_thunk+0xc/0x10
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
aisexec       S f75a5d98     0  3686      1
       f79c2530 00000086 00000246 f75a5d98 00000000 f4878000 00000246 c045f0c0 
       c0461d00 c0461d00 c0461d00 c01d720f f79c2684 c373ed00 f64a8a80 00000000 
       f75a5d88 00000005 00000000 f4878000 c01d720f 00000246 00000246 00000246 
Call Trace:
 [<c01d720f>] ipc_lock+0x60/0xb8
 [<c01d720f>] ipc_lock+0x60/0xb8
 [<c01d933f>] sys_semtimedop+0x5a8/0x7a4
 [<c01f4de3>] copy_from_user+0x2d/0x59
 [<c01f4de3>] copy_from_user+0x2d/0x59
 [<c02bede0>] verify_iovec+0x2a/0x8a
 [<c02b7d4d>] sys_sendmsg+0x232/0x237
 [<c011a66e>] __dequeue_entity+0x45/0xaa
 [<c011826e>] update_curr+0x40/0x52
 [<c011a6e7>] set_next_entity+0x14/0x38
 [<c011a8dc>] pick_next_task_fair+0x8f/0xa8
 [<c02b8c9e>] sys_socketcall+0xb7/0x2a9
 [<c0107401>] sys_ipc+0x6a/0x293
 [<c01f4a80>] trace_hardirqs_on_thunk+0xc/0x10
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
aisexec       S f75a5e98     0  3697      1
       f79d1710 00000086 00000246 f75a5e98 00000000 f4898000 00000246 c045f0c0 
       c0461d00 c0461d00 c0461d00 c01d720f f79d1864 c373ed00 f64a8a80 00000000 
       f75a5e88 00000006 00000000 f4898000 c01d720f 00000246 00000246 00000246 
Call Trace:
 [<c01d720f>] ipc_lock+0x60/0xb8
 [<c01d720f>] ipc_lock+0x60/0xb8
 [<c01d933f>] sys_semtimedop+0x5a8/0x7a4
 [<c01f4de3>] copy_from_user+0x2d/0x59
 [<c01f4de3>] copy_from_user+0x2d/0x59
 [<c02bede0>] verify_iovec+0x2a/0x8a
 [<c02b7d4d>] sys_sendmsg+0x232/0x237
 [<c02b8c9e>] sys_socketcall+0xb7/0x2a9
 [<c0107401>] sys_ipc+0x6a/0x293
 [<c01f4a80>] trace_hardirqs_on_thunk+0xc/0x10
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
aisexec       S f64f4998     0  3710      1
       f7155710 00000086 00000246 f64f4998 00000000 f50f4000 00000246 c045f0c0 
       c0461d00 c0461d00 c0461d00 c01d720f f7155864 c373ed00 f64a8a80 00000000 
       f64f4988 00000007 00000000 f50f4000 c01d720f 00000246 00000246 00000246 
Call Trace:
 [<c01d720f>] ipc_lock+0x60/0xb8
 [<c01d720f>] ipc_lock+0x60/0xb8
 [<c01d933f>] sys_semtimedop+0x5a8/0x7a4
 [<c011abae>] try_to_wake_up+0xa8/0x137
 [<c0117dce>] __wake_up_common+0x46/0x68
 [<c011a66e>] __dequeue_entity+0x45/0xaa
 [<c011a6e7>] set_next_entity+0x14/0x38
 [<c011a8dc>] pick_next_task_fair+0x8f/0xa8
 [<c0107401>] sys_ipc+0x6a/0x293
 [<c01f4a80>] trace_hardirqs_on_thunk+0xc/0x10
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
aisexec       S f64f4a98     0  3714      1
       f711b590 00000086 00000246 f64f4a98 00000000 f75d6000 00000246 c045f0c0 
       c0461d00 c0461d00 c0461d00 c01d720f f711b6e4 c3734d00 f64a8a80 00000000 
       f64f4a88 00000008 00000000 f75d6000 c01d720f 00000246 00000246 00000246 
Call Trace:
 [<c01d720f>] ipc_lock+0x60/0xb8
 [<c01d720f>] ipc_lock+0x60/0xb8
 [<c01d933f>] sys_semtimedop+0x5a8/0x7a4
 [<c0107401>] sys_ipc+0x6a/0x293
 [<c01f4a80>] trace_hardirqs_on_thunk+0xc/0x10
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
groupd        S f6465bf4     0  3527      1
       f6a59610 00000082 00000002 f6465bf4 00000002 00000000 f7abc4b0 c045f0c0 
       c0461d00 c0461d00 c0461d00 f6465bf8 f6a59764 c373ed00 f7963580 f6465efc 
       00000046 00000000 00000002 00000001 f4ca931c 00000286 f78c2080 f6465efc 
Call Trace:
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c0317dc6>] schedule_timeout+0x69/0xa1
 [<c030cb2e>] unix_poll+0x1a/0xab
 [<c017c62b>] do_sys_poll+0x238/0x311
 [<c017cfbf>] __pollwait+0x0/0xba
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011abae>] try_to_wake_up+0xa8/0x137
 [<c0117dce>] __wake_up_common+0x46/0x68
 [<c0319826>] _read_unlock+0x14/0x1c
 [<c030d03b>] unix_stream_sendmsg+0x1c3/0x32b
 [<c02b6f28>] sock_aio_write+0xcb/0xe8
 [<c02b740e>] sockfd_lookup_light+0x24/0x41
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c01a4b6f>] dnotify_parent+0x1f/0x5f
 [<c0171540>] vfs_write+0xe5/0x118
 [<c017c731>] sys_poll+0x2d/0x71
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
fenced        S f5177bf4     0  3532      1
       f6a48430 00000082 00000002 f5177bf4 f5177be8 00000000 f55f982c c045f0c0 
       c0461d00 c0461d00 c0461d00 f5177bf8 f6a48584 c373ed00 f78a5800 f5177ebc 
       ffff95a0 00000000 00000002 00000001 ffffffff 00000286 00000000 00000000 
Call Trace:
 [<c0317dc6>] schedule_timeout+0x69/0xa1
 [<c030cb2e>] unix_poll+0x1a/0xab
 [<c017c62b>] do_sys_poll+0x238/0x311
 [<c017cfbf>] __pollwait+0x0/0xba
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c0136043>] sched_clock_cpu+0x10f/0x158
 [<c011826e>] update_curr+0x40/0x52
 [<c011abae>] try_to_wake_up+0xa8/0x137
 [<c0117dce>] __wake_up_common+0x46/0x68
 [<c030c333>] unix_stream_recvmsg+0x3a0/0x4d8
 [<c02b7020>] sock_aio_read+0xdb/0xef
 [<c02b740e>] sockfd_lookup_light+0x24/0x41
 [<c02bce78>] skb_dequeue+0x50/0x56
 [<c0181530>] dput+0x97/0x145
 [<c0185904>] mntput_no_expire+0x18/0xef
 [<c016f61c>] filp_close+0x3e/0x62
 [<c017c731>] sys_poll+0x2d/0x71
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
dlm_controld  S c373baa0     0  3536      1
       f7abc4b0 00000082 000007fb c373baa0 00000002 00000000 f55faaac c045f0c0 
       c0461d00 c0461d00 c0461d00 f4837bf8 f7abc604 c373ed00 f756c800 f4837ebc 
       00000046 00000000 00000002 00000001 f55faa9c 00000286 f79de080 f4837ebc 
Call Trace:
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c0317dc6>] schedule_timeout+0x69/0xa1
 [<c030cb2e>] unix_poll+0x1a/0xab
 [<c017c62b>] do_sys_poll+0x238/0x311
 [<c017cfbf>] __pollwait+0x0/0xba
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c02d328f>] netlink_recvmsg+0x1d1/0x294
 [<c0117dce>] __wake_up_common+0x46/0x68
 [<c02b7a19>] sock_recvmsg+0xcf/0xf3
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c02b7020>] sock_aio_read+0xdb/0xef
 [<c02b740e>] sockfd_lookup_light+0x24/0x41
 [<c0170e16>] do_sync_read+0xd2/0x10e
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c02b88c1>] sys_recv+0x37/0x3b
 [<c02b8d90>] sys_socketcall+0x1a9/0x2a9
 [<c017c731>] sys_poll+0x2d/0x71
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
gfs_controld  S f75b1bf4     0  3540      1
       f7ac6da0 00000082 00000002 f75b1bf4 f75b1be8 00000000 00000000 c045f0c0 
       c0461d00 c0461d00 c0461d00 f75b1bf8 f7ac6ef4 c3734d00 f78a5080 f7ac6da0 
       ffffef62 00000000 f7b98a80 f75b1ed4 ffffffff f8939698 00000000 00000000 
Call Trace:
 [<c0317dc6>] schedule_timeout+0x69/0xa1
 [<c0319867>] _spin_unlock+0x14/0x1c
 [<c017c62b>] do_sys_poll+0x238/0x311
 [<c017cfbf>] __pollwait+0x0/0xba
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c01d720f>] ipc_lock+0x60/0xb8
 [<c031995f>] _spin_lock+0x31/0x3c
 [<c01d9056>] sys_semtimedop+0x2bf/0x7a4
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c0319bed>] _spin_lock_irq+0x37/0x42
 [<c0140d19>] futex_wake+0x52/0xe1
 [<c0141b58>] do_futex+0x7a/0x951
 [<c02c473c>] net_rx_action+0x9d/0x1db
 [<c011377d>] do_page_fault+0xa5/0x848
 [<c01424b5>] sys_futex+0x86/0x111
 [<c01f5002>] copy_to_user+0x2f/0x4d
 [<c017c731>] sys_poll+0x2d/0x71
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
clvmd         S f50fbb14     0  3644      1
       f71c81b0 00000086 00000002 f50fbb14 f50fbb08 00000000 c04b9e00 c045f0c0 
       c0461d00 c0461d00 c0461d00 f50fbb18 f71c8304 c3734d00 f64a8300 c04b9e00 
       ffffde5e 00000000 00000286 00000286 ffffffff c04b9e00 00000000 00000000 
Call Trace:
 [<c0317da1>] schedule_timeout+0x44/0xa1
 [<c01287ec>] process_timeout+0x0/0x5
 [<c017cbae>] do_select+0x3ef/0x4cb
 [<c017c7d3>] do_select+0x14/0x4cb
 [<c017cfbf>] __pollwait+0x0/0xba
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c0319c5a>] _spin_unlock_irq+0x20/0x23
 [<c011c3d4>] finish_task_switch+0x58/0xab
 [<c011c3a2>] finish_task_switch+0x26/0xab
 [<c03177b2>] schedule+0x228/0x708
 [<c0136043>] sched_clock_cpu+0x10f/0x158
 [<c011826e>] update_curr+0x40/0x52
 [<c011abae>] try_to_wake_up+0xa8/0x137
 [<c0117dce>] __wake_up_common+0x46/0x68
 [<c030c333>] unix_stream_recvmsg+0x3a0/0x4d8
 [<c017ce7d>] core_sys_select+0x1f3/0x335
 [<c017ccac>] core_sys_select+0x22/0x335
 [<c02b7020>] sock_aio_read+0xdb/0xef
 [<c02b740e>] sockfd_lookup_light+0x24/0x41
 [<c02b8888>] sys_recvfrom+0x11f/0x121
 [<c0319c5a>] _spin_unlock_irq+0x20/0x23
 [<c011c3d4>] finish_task_switch+0x58/0xab
 [<c011c3a2>] finish_task_switch+0x26/0xab
 [<c03177b2>] schedule+0x228/0x708
 [<c01a4b6f>] dnotify_parent+0x1f/0x5f
 [<c017d14f>] sys_select+0xd6/0x19e
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
clvmd         S 00000002     0  3655      1
       f7aa5490 00000086 f7aa5490 00000002 00000000 f75e02a0 f7aa5490 c045f0c0 
       c0461d00 c0461d00 c0461d00 00000046 f7aa55e4 c373ed00 f64a8300 f75e02d8 
       00000282 f75e02d8 b7dc3368 c0319cc1 00000046 00000246 00000246 f75e0290 
Call Trace:
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<f892ef52>] device_read+0x12d/0x347 [dlm]
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<f892ee25>] device_read+0x0/0x347 [dlm]
 [<c01715f8>] vfs_read+0x85/0x11b
 [<c017192b>] sys_read+0x41/0x6a
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
clvmd         S 00000002     0  3656      1
       f7ab3510 00000096 f7ab3510 00000002 00000000 c082d818 f7ab3510 c045f0c0 
       c0461d00 c0461d00 c0461d00 00000046 f7ab3664 c3734d00 f64a8300 f4913e3c 
       00000282 00000000 080b1fa4 c0319cc1 00000046 00000046 f4913e3c 00000282 
Call Trace:
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c0140c19>] futex_wait+0x318/0x396
 [<c0140d19>] futex_wake+0x52/0xe1
 [<c0319867>] _spin_unlock+0x14/0x1c
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c0141b9a>] do_futex+0xbc/0x951
 [<c01424b5>] sys_futex+0x86/0x111
 [<c0134e4d>] up_write+0x14/0x28
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
dlm_astd      S 00000002     0  3645      2
       f7aa43b0 00000092 00000246 00000002 00000000 f8939428 f7aa43b0 c045f0c0 
       c0461d00 c0461d00 c0461d00 00000246 f7aa4504 c3734d00 f7801800 00000246 
       00000000 00000002 00000046 00000046 f8939440 00000000 f8939440 f8939460 
Call Trace:
 [<f8a44dde>] gdlm_bast+0x0/0xa4 [gfs]
 [<f891f0f1>] dlm_astd+0x62/0x155 [dlm]
 [<f891f08f>] dlm_astd+0x0/0x155 [dlm]
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
dlm_scand     R running      0  3646      2
       f7abcd20 00000092 00000002 f50e7f88 f50e7f7c 00000000 f706e000 c045f0c0 
       c0461d00 c0461d00 c0461d00 f50e7f8c f7abce74 c373ed00 f64a8a80 f706e000 
       ffffef4b 00000000 00000282 00000282 ffffffff f706e000 00000000 00000000 
Call Trace:
 [<f892882a>] dlm_scand+0x0/0x77 [dlm]
 [<c0317da1>] schedule_timeout+0x44/0xa1
 [<c01287ec>] process_timeout+0x0/0x5
 [<f8928893>] dlm_scand+0x69/0x77 [dlm]
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
dlm_recv/0    R running      0  3647      2
       f78e5690 00000082 00000046 00000000 00000002 00000001 00000046 c045f0c0 
       c0461d00 c0461d00 c0461d00 00000000 f78e57e4 c3734d00 f7801800 00000286 
       00000001 f75e071c c0319cc1 00000000 00000046 00000046 f75e0724 00000286 
Call Trace:
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c012ee03>] worker_thread+0x0/0x89
 [<c012ee7d>] worker_thread+0x7a/0x89
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c012ee03>] worker_thread+0x0/0x89
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
dlm_recv/1    S 00000000     0  3648      2
       f6a58da0 00000082 00000046 00000000 00000002 00000001 00000046 c045f0c0 
       c0461d00 c0461d00 c0461d00 00000000 f6a58ef4 c373ed00 f756cd00 00000286 
       00000001 f75e069c c0319cc1 00000000 00000046 00000046 f75e06a4 00000286 
Call Trace:
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c012ee03>] worker_thread+0x0/0x89
 [<c012ee7d>] worker_thread+0x7a/0x89
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c012ee03>] worker_thread+0x0/0x89
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
dlm_send      S f4823f94     0  3649      2
       f6ab0aa0 00000082 00000002 f4823f94 f4823f88 00000000 00000046 c045f0c0 
       c0461d00 c0461d00 c0461d00 f4823f98 f6ab0bf4 c3734d00 f7801800 00000286 
       ffffef69 00000000 c0319cc1 00000000 ffffffff 00000046 00000000 00000000 
Call Trace:
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c012ee03>] worker_thread+0x0/0x89
 [<c012ee7d>] worker_thread+0x7a/0x89
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c012ee03>] worker_thread+0x0/0x89
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
dlm_recoverd  S f5dc1f80     0  3650      2
       f79f3290 00000086 00000002 f5dc1f80 f5dc1f74 00000000 00002bd4 c045f0c0 
       c0461d00 c0461d00 c0461d00 f5dc1f84 f79f33e4 c3734d00 f64a8a80 00000002 
       ffff97ae 00000000 f79f3290 00000046 ffffffff f7001280 00000000 00000000 
Call Trace:
 [<f892dd8d>] dlm_recoverd+0x163/0x55a [dlm]
 [<c0117dce>] __wake_up_common+0x46/0x68
 [<f892dc2a>] dlm_recoverd+0x0/0x55a [dlm]
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
kdmflush      S 00000000     0  3663      2
       f6ab82b0 00000082 00000046 00000000 00000002 00000001 00000046 c045f0c0 
       c0461d00 c0461d00 c0461d00 00000000 f6ab8404 c373ed00 f7c60d00 00000286 
       00000001 f482bd9c c0319cc1 00000000 00000046 00000046 f482bda4 00000286 
Call Trace:
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c012ee03>] worker_thread+0x0/0x89
 [<c012ee7d>] worker_thread+0x7a/0x89
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c012ee03>] worker_thread+0x0/0x89
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
kdmflush      S f6047f94     0  3669      2
       f71d0230 00000082 00000002 f6047f94 f6047f88 00000000 00000046 c045f0c0 
       c0461d00 c0461d00 c0461d00 f6047f98 f71d0384 c373ed00 f756c800 00000286 
       ffff980b 00000000 c0319cc1 00000000 ffffffff 00000046 00000000 00000000 
Call Trace:
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c012ee03>] worker_thread+0x0/0x89
 [<c012ee7d>] worker_thread+0x7a/0x89
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c012ee03>] worker_thread+0x0/0x89
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
lock_dlm1     S f5195f50     0  3690      2
       f79c2da0 00000086 00000002 f5195f50 f5195f44 00000000 f79c2da0 c045f0c0 
       c0461d00 c0461d00 c0461d00 f5195f54 f79c2ef4 c373ed00 f7896300 00000246 
       ffffa9f2 00000000 00000000 00000000 ffffffff 00000246 00000000 00000000 
Call Trace:
 [<f8a45d3e>] gdlm_thread+0x112/0x74a [gfs]
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<f8a4637d>] gdlm_thread1+0x0/0xa [gfs]
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
lock_dlm2     S 00000002     0  3694      2
       f7a022b0 00000086 c040dc00 00000002 00000000 f78d2b64 f7a022b0 c045f0c0 
       c0461d00 c0461d00 c0461d00 00000246 f7a02404 c3734d00 f78a5580 00000246 
       00000000 00000002 00000000 00000000 f8a45d59 00000246 00000246 f78d2b54 
Call Trace:
 [<f8a45d59>] gdlm_thread+0x12d/0x74a [gfs]
 [<f8a45d3e>] gdlm_thread+0x112/0x74a [gfs]
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<f8a46376>] gdlm_thread2+0x0/0x7 [gfs]
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
dlm_recoverd  S f8939428     0  3695      2
       f7a428a0 00000086 00000000 f8939428 f7a428a0 f5052bf4 00002bd4 c045f0c0 
       c0461d00 c0461d00 c0461d00 f5181800 f7a429f4 c373ed00 f64a8a80 00000002 
       00000000 00000000 f7a428a0 00000046 00000000 f7001280 f704b000 f7001280 
Call Trace:
 [<f892dd8d>] dlm_recoverd+0x163/0x55a [dlm]
 [<c0117dce>] __wake_up_common+0x46/0x68
 [<f892dc2a>] dlm_recoverd+0x0/0x55a [dlm]
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
gfs_scand     R running      0  3701      2
       f7154ea0 00000086 00000046 00000000 00000002 00000001 c04b9e00 c045f0c0 
       c0461d00 c0461d00 c0461d00 00000000 f7154ff4 c3734d00 f78a5080 c04b9e00 
       00000286 c04b9e00 00000286 00000286 f4827fa4 c04b9e00 c0128a5a 00000000 
Call Trace:
 [<c0128a5a>] __mod_timer+0x94/0xa3
 [<c0317da1>] schedule_timeout+0x44/0xa1
 [<f8a2e907>] gfs_scand+0x1b/0x48 [gfs]
 [<c01287ec>] process_timeout+0x0/0x5
 [<f8a2e925>] gfs_scand+0x39/0x48 [gfs]
 [<f8a2e8ec>] gfs_scand+0x0/0x48 [gfs]
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
gfs_glockd    S f4897f94     0  3702      2
       f7abd590 00000082 00000002 f4897f94 f4897f88 00000000 00000046 c045f0c0 
       c0461d00 c0461d00 c0461d00 f4897f98 f7abd6e4 c373ed00 f64a8a80 00000286 
       ffffef4b 00000000 c0319cc1 00000000 ffffffff 00000046 00000000 00000000 
Call Trace:
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<f8a2e8b8>] gfs_glockd+0x80/0xb4 [gfs]
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<f8a2e838>] gfs_glockd+0x0/0xb4 [gfs]
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
gfs_recoverd  S f5161f80     0  3704      2
       f7a17490 00000086 00000002 f5161f80 f5161f74 00000000 c04b9e00 c045f0c0 
       c0461d00 c0461d00 c0461d00 f5161f84 f7a175e4 c3734d00 f78a5580 c04b9e00 
       ffffde92 00000000 00000286 00000286 ffffffff c04b9e00 00000000 00000000 
Call Trace:
 [<c0317da1>] schedule_timeout+0x44/0xa1
 [<f8a2e94f>] gfs_recoverd+0x1b/0x48 [gfs]
 [<c01287ec>] process_timeout+0x0/0x5
 [<f8a2e96d>] gfs_recoverd+0x39/0x48 [gfs]
 [<f8a2e934>] gfs_recoverd+0x0/0x48 [gfs]
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
gfs_logd      R running      0  3705      2
       f7154630 00000082 00000400 c373baa0 00000002 f4893f2c f706e000 c045f0c0 
       c0461d00 c0461d00 c0461d00 f4893f34 f7154784 c373ed00 f781a580 f706e000 
       ffffef77 00000000 00000286 00000286 ffffffff f706e000 00000000 00000000 
Call Trace:
 [<c0317da1>] schedule_timeout+0x44/0xa1
 [<f8a2e99f>] gfs_logd+0x23/0xb7 [gfs]
 [<c01287ec>] process_timeout+0x0/0x5
 [<f8a2e9bd>] gfs_logd+0x41/0xb7 [gfs]
 [<f8a2e97c>] gfs_logd+0x0/0xb7 [gfs]
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
gfs_quotad    R running      0  3706      2
       f71e0ba0 00000086 00000002 f6039f64 f6039f58 00000000 c04b9e00 c045f0c0 
       c0461d00 c0461d00 c0461d00 f6039f68 f71e0cf4 c3734d00 f7c60300 c04b9e00 
       ffffee49 00000000 00000282 00000282 ffffffff c04b9e00 00000000 00000000 
Call Trace:
 [<c0317da1>] schedule_timeout+0x44/0xa1
 [<f8a2eabb>] gfs_quotad+0x88/0x16d [gfs]
 [<c01287ec>] process_timeout+0x0/0x5
 [<f8a2ead9>] gfs_quotad+0xa6/0x16d [gfs]
 [<f8a2ea33>] gfs_quotad+0x0/0x16d [gfs]
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
gfs_inoded    R running      0  3707      2
       f71e0330 00000086 00000002 f6151f80 f6151f74 00000000 c04b9e00 c045f0c0 
       c0461d00 c0461d00 c0461d00 f6151f84 f71e0484 c3734d00 f78a5580 c04b9e00 
       ffffea63 00000000 00000286 00000286 ffffffff c04b9e00 00000000 00000000 
Call Trace:
 [<c0317da1>] schedule_timeout+0x44/0xa1
 [<f8a2e80b>] gfs_inoded+0x1b/0x48 [gfs]
 [<c01287ec>] process_timeout+0x0/0x5
 [<f8a2e829>] gfs_inoded+0x39/0x48 [gfs]
 [<f8a2e7f0>] gfs_inoded+0x0/0x48 [gfs]
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
lock_dlm1     S 00000002     0  3711      2
       f71c8a20 00000086 00000000 00000002 00000000 f71bb764 f71c8a20 c045f0c0 
       c0461d00 c0461d00 c0461d00 00000246 f71c8b74 c3734d00 f7801800 00000246 
       00000000 00000002 00000000 00000000 f8a45d59 00000246 00000246 f71bb754 
Call Trace:
 [<f8a45d59>] gdlm_thread+0x12d/0x74a [gfs]
 [<f8a45d3e>] gdlm_thread+0x112/0x74a [gfs]
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<f8a4637d>] gdlm_thread1+0x0/0xa [gfs]
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
lock_dlm2     S f6157f50     0  3712      2
       f6a68ea0 00000086 00000002 f6157f50 f6157f44 00000000 f6a68ea0 c045f0c0 
       c0461d00 c0461d00 c0461d00 f6157f54 f6a68ff4 c373ed00 f781a580 00000246 
       ffffef73 00000000 00000000 00000000 ffffffff 00000246 00000000 00000000 
Call Trace:
 [<f8a45d3e>] gdlm_thread+0x112/0x74a [gfs]
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<f8a46376>] gdlm_thread2+0x0/0x7 [gfs]
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
dlm_recoverd  S f8939428     0  3713      2
       f6a69710 00000086 00000000 f8939428 f6a69710 f518ebf4 00002bd4 c045f0c0 
       c0461d00 c0461d00 c0461d00 f64b5800 f6a69864 c373ed00 f7801580 00000002 
       00000000 00000000 f6a69710 00000046 00000000 f7001280 f704b000 f7001280 
Call Trace:
 [<f892dd8d>] dlm_recoverd+0x163/0x55a [dlm]
 [<c0117dce>] __wake_up_common+0x46/0x68
 [<f892dc2a>] dlm_recoverd+0x0/0x55a [dlm]
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
gfs_scand     R running      0  3719      2
       f71580b0 00000086 00000046 00000000 00000002 00000001 c04b9e00 c045f0c0 
       c0461d00 c0461d00 c0461d00 00000000 f7158204 c3734d00 f78a5a80 c04b9e00 
       00000286 c04b9e00 00000286 00000286 f74b5fa4 c04b9e00 c0128a5a 00000000 
Call Trace:
 [<c0128a5a>] __mod_timer+0x94/0xa3
 [<c0317da1>] schedule_timeout+0x44/0xa1
 [<f8a2e907>] gfs_scand+0x1b/0x48 [gfs]
 [<c01287ec>] process_timeout+0x0/0x5
 [<f8a2e925>] gfs_scand+0x39/0x48 [gfs]
 [<f8a2e8ec>] gfs_scand+0x0/0x48 [gfs]
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
gfs_glockd    S f219112c     0  3720      2
       f6ab1310 00000082 f2191158 f219112c 00000246 f8b44994 00000046 c045f0c0 
       c0461d00 c0461d00 c0461d00 00000000 f6ab1464 c3734d00 f7801800 00000286 
       00000001 f74b7fbc c0319cc1 00000000 00000046 00000046 f8b83504 00000286 
Call Trace:
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<f8a2e8b8>] gfs_glockd+0x80/0xb4 [gfs]
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<f8a2e838>] gfs_glockd+0x0/0xb4 [gfs]
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
gfs_recoverd  S f4b9bf80     0  3722      2
       f79b0ca0 00000086 00000002 f4b9bf80 f4b9bf74 00000000 c04b9e00 c045f0c0 
       c0461d00 c0461d00 c0461d00 f4b9bf84 f79b0df4 c3734d00 f64a8a80 c04b9e00 
       ffffdeeb 00000000 00000286 00000286 ffffffff c04b9e00 00000000 00000000 
Call Trace:
 [<c0317da1>] schedule_timeout+0x44/0xa1
 [<f8a2e94f>] gfs_recoverd+0x1b/0x48 [gfs]
 [<c01287ec>] process_timeout+0x0/0x5
 [<f8a2e96d>] gfs_recoverd+0x39/0x48 [gfs]
 [<f8a2e934>] gfs_recoverd+0x0/0x48 [gfs]
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
gfs_logd      R running      0  3723      2
       f79b0430 00000082 00000002 f4b9df30 f4b9df24 00000000 f706e000 c045f0c0 
       c0461d00 c0461d00 c0461d00 f4b9df34 f79b0584 c373ed00 f64a8a80 f706e000 
       ffffef4b 00000000 00000286 00000286 ffffffff f706e000 00000000 00000000 
Call Trace:
 [<c0317da1>] schedule_timeout+0x44/0xa1
 [<f8a2e99f>] gfs_logd+0x23/0xb7 [gfs]
 [<c01287ec>] process_timeout+0x0/0x5
 [<f8a2e9bd>] gfs_logd+0x41/0xb7 [gfs]
 [<f8a2e97c>] gfs_logd+0x0/0xb7 [gfs]
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
gfs_quotad    R running      0  3724      2
       f6a58530 00000086 00000002 f4b9ff64 f4b9ff58 00000000 c04b9e00 c045f0c0 
       c0461d00 c0461d00 c0461d00 f4b9ff68 f6a58684 c3734d00 f7c60300 c04b9e00 
       ffffee87 00000000 00000282 00000282 ffffffff c04b9e00 00000000 00000000 
Call Trace:
 [<c0317da1>] schedule_timeout+0x44/0xa1
 [<f8a2eabb>] gfs_quotad+0x88/0x16d [gfs]
 [<c01287ec>] process_timeout+0x0/0x5
 [<f8a2ead9>] gfs_quotad+0xa6/0x16d [gfs]
 [<f8a2ea33>] gfs_quotad+0x0/0x16d [gfs]
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
gfs_inoded    R running      0  3725      2
       f7a749a0 00000086 00000002 f4ba1f80 f4ba1f74 00000000 c04b9e00 c045f0c0 
       c0461d00 c0461d00 c0461d00 f4ba1f84 f7a74af4 c3734d00 f781a300 c04b9e00 
       ffffeab3 00000000 00000286 00000286 ffffffff c04b9e00 00000000 00000000 
Call Trace:
 [<c0317da1>] schedule_timeout+0x44/0xa1
 [<f8a2e80b>] gfs_inoded+0x1b/0x48 [gfs]
 [<c01287ec>] process_timeout+0x0/0x5
 [<f8a2e829>] gfs_inoded+0x39/0x48 [gfs]
 [<f8a2e7f0>] gfs_inoded+0x0/0x48 [gfs]
 [<c0131378>] kthread+0x34/0x55
 [<c0131344>] kthread+0x0/0x55
 [<c010469b>] kernel_thread_helper+0x7/0x1c
 =======================
lighttpd      R running      0  3859      1
       f79a5490 00200082 00000002 f2889bf4 f2889be8 00000000 f706e000 c045f0c0 
       c0461d00 c0461d00 c0461d00 f2889bf8 f79a55e4 c373ed00 f7c60a80 f706e000 
       ffffef2f 00000000 00200282 00200282 ffffffff f706e000 00000000 00000000 
Call Trace:
 [<c0317da1>] schedule_timeout+0x44/0xa1
 [<c01287ec>] process_timeout+0x0/0x5
 [<c017c62b>] do_sys_poll+0x238/0x311
 [<c017cfbf>] __pollwait+0x0/0xba
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c02d0eae>] __qdisc_run+0x7f/0x1c5
 [<c031995f>] _spin_lock+0x31/0x3c
 [<c02d0eae>] __qdisc_run+0x7f/0x1c5
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c025a9b9>] scsi_next_command+0x25/0x2f
 [<c025ab90>] scsi_end_request+0x52/0x75
 [<c025b6dc>] scsi_io_completion+0x87/0x3de
 [<c031995f>] _spin_lock+0x31/0x3c
 [<c01e3113>] blk_done_softirq+0x5b/0x68
 [<c0124797>] __do_softirq+0xa2/0xf9
 [<c0124699>] _local_bh_enable+0x44/0xa0
 [<c017c731>] sys_poll+0x2d/0x71
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
atd           S f3de9f2c     0  3866      1
       f79ca5b0 00000082 00000002 f3de9f2c f3de9f20 00000000 f6455ddc c045f0c0 
       c0461d00 c0461d00 c0461d00 f3de9f30 f79ca704 c3734d00 f7832800 f3de9f60 
       ffff9c98 00000000 0000000a c045ef20 ffffffff 00000350 00000000 00000000 
Call Trace:
 [<c0318a6b>] do_nanosleep+0x6d/0x96
 [<c01348df>] hrtimer_nanosleep+0x50/0xad
 [<c0134376>] hrtimer_wakeup+0x0/0x18
 [<c0134994>] sys_nanosleep+0x58/0x5c
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
cron          S f29dff2c     0  3877      1
       f6a504b0 00000082 00000002 f29dff2c f29dff20 00000000 c04782ec c045f0c0 
       c0461d00 c0461d00 c0461d00 f29dff30 f6a50604 c3734d00 f7963080 f29dff60 
       ffffe623 00000000 00000036 c045ef20 ffffffff 00000044 00000000 00000000 
Call Trace:
 [<c0318a6b>] do_nanosleep+0x6d/0x96
 [<c01348df>] hrtimer_nanosleep+0x50/0xad
 [<c0134376>] hrtimer_wakeup+0x0/0x18
 [<c0134994>] sys_nanosleep+0x58/0x5c
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
S99phplog     R running      0  3887      1
       f79cae20 00000082 00000002 f29eff2c f29eff20 00000000 f64189dc c045f0c0 
       c0461d00 c0461d00 c0461d00 f29eff30 f79caf74 c373ed00 f78ff800 f29eff60 
       ffffe5ac 00000000 00000036 c045ef20 ffffffff 0000003d 00000000 00000000 
Call Trace:
 [<c0318a6b>] do_nanosleep+0x6d/0x96
 [<c01348df>] hrtimer_nanosleep+0x50/0xad
 [<c0134376>] hrtimer_wakeup+0x0/0x18
 [<c0134994>] sys_nanosleep+0x58/0x5c
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
lighttpd      R running      0  3937      1
       f71d82b0 00200082 00000002 f2825bf4 f2825be8 00000000 c04b9e00 c045f0c0 
       c0461d00 c0461d00 c0461d00 f2825bf8 f71d8404 c3734d00 f78a5580 c04b9e00 
       ffffef43 00000000 00200282 00200282 ffffffff c04b9e00 00000000 00000000 
Call Trace:
 [<c0317da1>] schedule_timeout+0x44/0xa1
 [<c01287ec>] process_timeout+0x0/0x5
 [<c017c62b>] do_sys_poll+0x238/0x311
 [<c017cfbf>] __pollwait+0x0/0xba
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<f88ed512>] e1000_xmit_frame+0x7a5/0xb7b [e1000e]
 [<c01248bb>] local_bh_enable_ip+0x79/0xb5
 [<f896e4b8>] ipt_do_table+0x1fe/0x496 [ip_tables]
 [<c02d0eae>] __qdisc_run+0x7f/0x1c5
 [<c031995f>] _spin_lock+0x31/0x3c
 [<c02d0eae>] __qdisc_run+0x7f/0x1c5
 [<c02c5099>] dev_queue_xmit+0xa2/0x509
 [<c02c50e5>] dev_queue_xmit+0xee/0x509
 [<c0124e57>] local_bh_enable+0x7f/0xc8
 [<c02c50e5>] dev_queue_xmit+0xee/0x509
 [<c02c502d>] dev_queue_xmit+0x36/0x509
 [<c02dee66>] ip_finish_output+0x112/0x26b
 [<c02de9da>] ip_local_out+0x15/0x17
 [<c02df4e2>] ip_queue_xmit+0x195/0x327
 [<c010659e>] do_IRQ+0x40/0x72
 [<c014ddda>] __generic_file_aio_write_nolock+0x285/0x4df
 [<c01f4a80>] trace_hardirqs_on_thunk+0xc/0x10
 [<c0103aa8>] restore_nocheck_notrace+0x0/0xe
 [<c02f3f65>] tcp_v4_send_check+0x3b/0xc8
 [<c02ef055>] tcp_transmit_skb+0x363/0x659
 [<c014e303>] generic_file_aio_write+0x73/0xd8
 [<c0181530>] dput+0x97/0x145
 [<c0185904>] mntput_no_expire+0x18/0xef
 [<c016f61c>] filp_close+0x3e/0x62
 [<c017c731>] sys_poll+0x2d/0x71
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
php5-cgi      S 00000002     0  3944      1
       f7ac6530 00000082 f7ac6530 00000002 00000000 c040c990 f7ac6530 c045f0c0 
       c0461d00 c0461d00 c0461d00 00000246 f7ac6684 c3734d00 f78ff300 00000000 
       00000002 00000002 00000000 00000000 c01228e1 00000246 00000246 c040c980 
Call Trace:
 [<c01228e1>] do_wait+0x96/0x342
 [<c0122ad7>] do_wait+0x28c/0x342
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c0122bf6>] sys_wait4+0x69/0xa4
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
php5-cgi      S 00000002     0  3956      1
       f7a821b0 00000082 f7a821b0 00000002 00000000 c040c990 f7a821b0 c045f0c0 
       c0461d00 c0461d00 c0461d00 00000246 f7a82304 c3734d00 f78ff080 00000000 
       00000002 00000002 00000000 00000000 c01228e1 00000246 00000246 c040c980 
Call Trace:
 [<c01228e1>] do_wait+0x96/0x342
 [<c0122ad7>] do_wait+0x28c/0x342
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c0122bf6>] sys_wait4+0x69/0xa4
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
php5-cgi      S f2847f24     0  3968      1
       f79c3610 00000082 00000002 f2847f24 f2847f18 00000000 f79c3610 c045f0c0 
       c0461d00 c0461d00 c0461d00 f2847f28 f79c3764 c3734d00 f7967300 00000000 
       ffff9d3f 00000000 00000000 00000000 ffffffff 00000246 00000000 00000000 
Call Trace:
 [<c0122ad7>] do_wait+0x28c/0x342
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c0122bf6>] sys_wait4+0x69/0xa4
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
php5-cgi      S f41a9f24     0  3980      1
       f6a50d20 00000082 00000002 f41a9f24 f41a9f18 00000000 f6a50d20 c045f0c0 
       c0461d00 c0461d00 c0461d00 f41a9f28 f6a50e74 c373ed00 f6b40800 00000000 
       ffff9d57 00000000 00000000 00000000 ffffffff 00000246 00000000 00000000 
Call Trace:
 [<c0122ad7>] do_wait+0x28c/0x342
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c0122bf6>] sys_wait4+0x69/0xa4
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
php5-cgi      S f4117f24     0  3992      1
       f7a0aba0 00000082 00000002 f4117f24 f4117f18 00000000 f7a0aba0 c045f0c0 
       c0461d00 c0461d00 c0461d00 f4117f28 f7a0acf4 c3734d00 f6b40300 00000000 
       ffff9d5f 00000000 00000000 00000000 ffffffff 00000246 00000000 00000000 
Call Trace:
 [<c0122ad7>] do_wait+0x28c/0x342
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c0122bf6>] sys_wait4+0x69/0xa4
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
php5-cgi      S f402bf24     0  4005      1
       f70822b0 00000082 00000002 f402bf24 f402bf18 00000000 f70822b0 c045f0c0 
       c0461d00 c0461d00 c0461d00 f402bf28 f7082404 c3734d00 f7801a80 00000000 
       ffff9d3f 00000000 00000000 00000000 ffffffff 00000246 00000000 00000000 
Call Trace:
 [<c0122ad7>] do_wait+0x28c/0x342
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c0122bf6>] sys_wait4+0x69/0xa4
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
php5-cgi      S 00000002     0  4017      1
       f6aa81b0 00000082 f6aa81b0 00000002 00000000 c040c990 f6aa81b0 c045f0c0 
       c0461d00 c0461d00 c0461d00 00000246 f6aa8304 c3734d00 f7982d00 00000000 
       00000002 00000002 00000000 00000000 c01228e1 00000246 00000246 c040c980 
Call Trace:
 [<c01228e1>] do_wait+0x96/0x342
 [<c0122ad7>] do_wait+0x28c/0x342
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c0122bf6>] sys_wait4+0x69/0xa4
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
php5-cgi      S c373baa0     0  4019   3956
       f7a2e530 00000082 00000400 c373baa0 00000002 f29cbe20 0000000c c045f0c0 
       c0461d00 c0461d00 c0461d00 f29cbe28 f7a2e684 c373ed00 f7982a80 f759e45c 
       ffff9d55 00000000 00000001 f759e44c ffffffff 00000000 00000000 00000000 
Call Trace:
 [<c0317dc6>] schedule_timeout+0x69/0xa1
 [<c02bf79f>] __skb_recv_datagram+0x65/0x22d
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c02bf989>] skb_recv_datagram+0x22/0x27
 [<c030c4c1>] unix_accept+0x56/0xe3
 [<c02b8b03>] do_accept+0xf0/0x1d4
 [<c015bb55>] handle_mm_fault+0x176/0x8a4
 [<c011377d>] do_page_fault+0xa5/0x848
 [<c011382e>] do_page_fault+0x156/0x848
 [<c02b8e25>] sys_socketcall+0x23e/0x2a9
 [<c01f4a80>] trace_hardirqs_on_thunk+0xc/0x10
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
php5-cgi      S 001200d2     0  4020   3956
       f7998ba0 00000082 f7998ba0 001200d2 00000246 c03ed880 0000000c c045f0c0 
       c0461d00 c0461d00 c0461d00 f7998ba0 f7998cf4 c373ed00 f7982580 f759e45c 
       00000046 f4ca9d2c 00000001 f759e44c 00000046 00000000 00000002 00000001 
Call Trace:
 [<c0317dc6>] schedule_timeout+0x69/0xa1
 [<c02bf79f>] __skb_recv_datagram+0x65/0x22d
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c02bf989>] skb_recv_datagram+0x22/0x27
 [<c030c4c1>] unix_accept+0x56/0xe3
 [<c02b8b03>] do_accept+0xf0/0x1d4
 [<c015bb55>] handle_mm_fault+0x176/0x8a4
 [<c0134eca>] up_read+0x14/0x2a
 [<c011377d>] do_page_fault+0xa5/0x848
 [<c011382e>] do_page_fault+0x156/0x848
 [<c02b8e25>] sys_socketcall+0x23e/0x2a9
 [<c01f4a80>] trace_hardirqs_on_thunk+0xc/0x10
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
php5-cgi      S 0000001f     0  4021   3956
       f79e8130 00000082 c0405c80 0000001f 00000246 c03ed880 0000000c c045f0c0 
       c0461d00 c0461d00 c0461d00 f79e8130 f79e8284 c373ed00 f6505080 f759e45c 
       00000046 f4ca9d2c 00000001 f759e44c 00000046 00000000 00000002 00000001 
Call Trace:
 [<c0317dc6>] schedule_timeout+0x69/0xa1
 [<c02bf79f>] __skb_recv_datagram+0x65/0x22d
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c02bf989>] skb_recv_datagram+0x22/0x27
 [<c030c4c1>] unix_accept+0x56/0xe3
 [<c02b8b03>] do_accept+0xf0/0x1d4
 [<c015bb55>] handle_mm_fault+0x176/0x8a4
 [<c0134eca>] up_read+0x14/0x2a
 [<c011377d>] do_page_fault+0xa5/0x848
 [<c011382e>] do_page_fault+0x156/0x848
 [<c02b8e25>] sys_socketcall+0x23e/0x2a9
 [<c01f4a80>] trace_hardirqs_on_thunk+0xc/0x10
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
php5-cgi      S 001200d2     0  4022   3956
       f7a1aca0 00000082 00000000 001200d2 00000246 c03ed880 0000000c c045f0c0 
       c0461d00 c0461d00 c0461d00 f7a1aca0 f7a1adf4 c373ed00 f6505a80 f759e45c 
       00000046 f4ca9d2c 00000001 f759e44c 00000046 00000000 00000002 00000001 
Call Trace:
 [<c0317dc6>] schedule_timeout+0x69/0xa1
 [<c02bf79f>] __skb_recv_datagram+0x65/0x22d
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c02bf989>] skb_recv_datagram+0x22/0x27
 [<c030c4c1>] unix_accept+0x56/0xe3
 [<c02b8b03>] do_accept+0xf0/0x1d4
 [<c015bb55>] handle_mm_fault+0x176/0x8a4
 [<c0134eca>] up_read+0x14/0x2a
 [<c011377d>] do_page_fault+0xa5/0x848
 [<c011382e>] do_page_fault+0x156/0x848
 [<c02b8e25>] sys_socketcall+0x23e/0x2a9
 [<c01f4a80>] trace_hardirqs_on_thunk+0xc/0x10
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
php5-cgi      S 001200d2     0  4023   3944
       f7a2f610 00000082 00000000 001200d2 00000246 c03ed880 0000000c c045f0c0 
       c0461d00 c0461d00 c0461d00 f7a2f610 f7a2f764 c373ed00 f6505580 f782b75c 
       00000046 f36f932c 00000001 f782b74c 00000046 00000000 00000002 00000001 
Call Trace:
 [<c0317dc6>] schedule_timeout+0x69/0xa1
 [<c02bf79f>] __skb_recv_datagram+0x65/0x22d
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c02bf989>] skb_recv_datagram+0x22/0x27
 [<c030c4c1>] unix_accept+0x56/0xe3
 [<c02b8b03>] do_accept+0xf0/0x1d4
 [<c015bb55>] handle_mm_fault+0x176/0x8a4
 [<c011377d>] do_page_fault+0xa5/0x848
 [<c011382e>] do_page_fault+0x156/0x848
 [<c02b8e25>] sys_socketcall+0x23e/0x2a9
 [<c01f4a80>] trace_hardirqs_on_thunk+0xc/0x10
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
php5-cgi      S f2927e24     0  4024   3944
       f6a78920 00000082 00000002 f2927e24 f2927e18 00000000 00000000 c045f0c0 
       c0461d00 c0461d00 c0461d00 f2927e28 f6a78a74 c3734d00 f6505800 f782b75c 
       ffffb5a6 00000000 00000001 f782b74c ffffffff 00000000 00000000 00000000 
Call Trace:
 [<c0317dc6>] schedule_timeout+0x69/0xa1
 [<c02bf79f>] __skb_recv_datagram+0x65/0x22d
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c02bf989>] skb_recv_datagram+0x22/0x27
 [<c030c4c1>] unix_accept+0x56/0xe3
 [<c02b8b03>] do_accept+0xf0/0x1d4
 [<c02bce78>] skb_dequeue+0x50/0x56
 [<c0181530>] dput+0x97/0x145
 [<c02b8e25>] sys_socketcall+0x23e/0x2a9
 [<c016f61c>] filp_close+0x3e/0x62
 [<c01f4a80>] trace_hardirqs_on_thunk+0xc/0x10
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
php5-cgi      S c373baa0     0  4025   3944
       f79da8a0 00000082 00000400 c373baa0 00000002 f2907e20 000800d0 c045f0c0 
       c0461d00 c0461d00 c0461d00 f2907e28 f79da9f4 c373ed00 f781a080 f782b75c 
       ffff9d56 00000000 00000001 f782b74c ffffffff 00000000 00000000 00000000 
Call Trace:
 [<c0317dc6>] schedule_timeout+0x69/0xa1
 [<c02bf79f>] __skb_recv_datagram+0x65/0x22d
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c02bf989>] skb_recv_datagram+0x22/0x27
 [<c030c4c1>] unix_accept+0x56/0xe3
 [<c02b8b03>] do_accept+0xf0/0x1d4
 [<c015bb55>] handle_mm_fault+0x176/0x8a4
 [<c0134eca>] up_read+0x14/0x2a
 [<c011377d>] do_page_fault+0xa5/0x848
 [<c011382e>] do_page_fault+0x156/0x848
 [<c02b8e25>] sys_socketcall+0x23e/0x2a9
 [<c01f4a80>] trace_hardirqs_on_thunk+0xc/0x10
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
php5-cgi      S c373baa0     0  4026   3944
       f7a16c20 00000082 00000400 c373baa0 00000002 f28ade20 0000000c c045f0c0 
       c0461d00 c0461d00 c0461d00 f28ade28 f7a16d74 c373ed00 f781a800 f782b75c 
       ffff9d56 00000000 00000001 f782b74c ffffffff 00000000 00000000 00000000 
Call Trace:
 [<c0317dc6>] schedule_timeout+0x69/0xa1
 [<c02bf79f>] __skb_recv_datagram+0x65/0x22d
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c02bf989>] skb_recv_datagram+0x22/0x27
 [<c030c4c1>] unix_accept+0x56/0xe3
 [<c02b8b03>] do_accept+0xf0/0x1d4
 [<c015bb55>] handle_mm_fault+0x176/0x8a4
 [<c0134eca>] up_read+0x14/0x2a
 [<c011377d>] do_page_fault+0xa5/0x848
 [<c011382e>] do_page_fault+0x156/0x848
 [<c02b8e25>] sys_socketcall+0x23e/0x2a9
 [<c01f4a80>] trace_hardirqs_on_thunk+0xc/0x10
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
php5-cgi      S c01512af     0  4027   3968
       f7ab2ca0 00000082 c3423b1c c01512af 00000246 c03ed880 0000000c c045f0c0 
       c0461d00 c0461d00 c0461d00 f7ab2ca0 f7ab2df4 c3734d00 f781aa80 f782ba5c 
       00000046 f36f90ac 00000001 f782ba4c 00000046 00000000 00000002 00000001 
Call Trace:
 [<c01512af>] rmqueue_bulk+0x67/0x71
 [<c0317dc6>] schedule_timeout+0x69/0xa1
 [<c02bf79f>] __skb_recv_datagram+0x65/0x22d
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c02bf989>] skb_recv_datagram+0x22/0x27
 [<c030c4c1>] unix_accept+0x56/0xe3
 [<c02b8b03>] do_accept+0xf0/0x1d4
 [<c015bb55>] handle_mm_fault+0x176/0x8a4
 [<c0134eca>] up_read+0x14/0x2a
 [<c011377d>] do_page_fault+0xa5/0x848
 [<c011382e>] do_page_fault+0x156/0x848
 [<c02b8e25>] sys_socketcall+0x23e/0x2a9
 [<c01f4a80>] trace_hardirqs_on_thunk+0xc/0x10
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
php5-cgi      S 001200d2     0  4028   4005
       f7a962b0 00000082 00000000 001200d2 00000246 c03ed880 0000000c c045f0c0 
       c0461d00 c0461d00 c0461d00 f7a962b0 f7a96404 c373ed00 f6b40a80 f759ea5c 
       00000046 f4ca90ac 00000001 f759ea4c 00000046 00000000 00000002 00000001 
Call Trace:
 [<c0317dc6>] schedule_timeout+0x69/0xa1
 [<c02bf79f>] __skb_recv_datagram+0x65/0x22d
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c02bf989>] skb_recv_datagram+0x22/0x27
 [<c030c4c1>] unix_accept+0x56/0xe3
 [<c02b8b03>] do_accept+0xf0/0x1d4
 [<c015bb55>] handle_mm_fault+0x176/0x8a4
 [<c0134eca>] up_read+0x14/0x2a
 [<c011377d>] do_page_fault+0xa5/0x848
 [<c011382e>] do_page_fault+0x156/0x848
 [<c02b8e25>] sys_socketcall+0x23e/0x2a9
 [<c01f4a80>] trace_hardirqs_on_thunk+0xc/0x10
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
php5-cgi      S ffffffff     0  4029   4017
       f79b8d20 00000082 00000001 ffffffff 00000000 00000000 00000000 c045f0c0 
       c0461d00 c0461d00 c0461d00 f79b8d20 f79b8e74 c3734d00 f781ad00 f7855a5c 
       00000046 f241e82c 00000001 f7855a4c 00000046 00000000 00000002 00000001 
Call Trace:
 [<c0317dc6>] schedule_timeout+0x69/0xa1
 [<c02bf79f>] __skb_recv_datagram+0x65/0x22d
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c02bf989>] skb_recv_datagram+0x22/0x27
 [<c030c4c1>] unix_accept+0x56/0xe3
 [<c02b8b03>] do_accept+0xf0/0x1d4
 [<c02bce78>] skb_dequeue+0x50/0x56
 [<c0181530>] dput+0x97/0x145
 [<c02b8e25>] sys_socketcall+0x23e/0x2a9
 [<c016f61c>] filp_close+0x3e/0x62
 [<c01f4a80>] trace_hardirqs_on_thunk+0xc/0x10
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
php5-cgi      S f7054bb4     0  4030   3968
       f7a6a920 00000082 00000000 f7054bb4 f7a6a920 f7a6a920 000800d0 c045f0c0 
       c0461d00 c0461d00 c0461d00 f7a6a920 f7a6aa74 c3734d00 f7896a80 f782ba5c 
       00000046 f36f90ac 00000001 f782ba4c 00000046 00000000 00000002 00000001 
Call Trace:
 [<c0317dc6>] schedule_timeout+0x69/0xa1
 [<c02bf79f>] __skb_recv_datagram+0x65/0x22d
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c02bf989>] skb_recv_datagram+0x22/0x27
 [<c030c4c1>] unix_accept+0x56/0xe3
 [<c02b8b03>] do_accept+0xf0/0x1d4
 [<c015bb55>] handle_mm_fault+0x176/0x8a4
 [<c0134eca>] up_read+0x14/0x2a
 [<c011377d>] do_page_fault+0xa5/0x848
 [<c011382e>] do_page_fault+0x156/0x848
 [<c02b8e25>] sys_socketcall+0x23e/0x2a9
 [<c01f4a80>] trace_hardirqs_on_thunk+0xc/0x10
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
php5-cgi      S f2dcbe24     0  4031   4017
       f71d9390 00000082 00000002 f2dcbe24 f2dcbe18 00000000 00000000 c045f0c0 
       c0461d00 c0461d00 c0461d00 f2dcbe28 f71d94e4 c3734d00 f7896080 f7855a5c 
       ffffa048 00000000 00000001 f7855a4c ffffffff 00000000 00000000 00000000 
Call Trace:
 [<c0317dc6>] schedule_timeout+0x69/0xa1
 [<c02bf79f>] __skb_recv_datagram+0x65/0x22d
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c02bf989>] skb_recv_datagram+0x22/0x27
 [<c030c4c1>] unix_accept+0x56/0xe3
 [<c02b8b03>] do_accept+0xf0/0x1d4
 [<c02bce78>] skb_dequeue+0x50/0x56
 [<c0181530>] dput+0x97/0x145
 [<c02b8e25>] sys_socketcall+0x23e/0x2a9
 [<c016f61c>] filp_close+0x3e/0x62
 [<c01f4a80>] trace_hardirqs_on_thunk+0xc/0x10
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
php5-cgi      S c01512af     0  4032   3968
       f7a163b0 00000082 c3424e00 c01512af 00000246 c03ed880 0000000c c045f0c0 
       c0461d00 c0461d00 c0461d00 f7a163b0 f7a16504 c3734d00 f7c60d00 f782ba5c 
       00000046 f36f90ac 00000001 f782ba4c 00000046 00000000 00000002 00000001 
Call Trace:
 [<c01512af>] rmqueue_bulk+0x67/0x71
 [<c0317dc6>] schedule_timeout+0x69/0xa1
 [<c02bf79f>] __skb_recv_datagram+0x65/0x22d
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c02bf989>] skb_recv_datagram+0x22/0x27
 [<c030c4c1>] unix_accept+0x56/0xe3
 [<c02b8b03>] do_accept+0xf0/0x1d4
 [<c015bb55>] handle_mm_fault+0x176/0x8a4
 [<c0134eca>] up_read+0x14/0x2a
 [<c011377d>] do_page_fault+0xa5/0x848
 [<c011382e>] do_page_fault+0x156/0x848
 [<c02b8e25>] sys_socketcall+0x23e/0x2a9
 [<c01f4a80>] trace_hardirqs_on_thunk+0xc/0x10
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
php5-cgi      S f4177e24     0  4033   4017
       f7a6a0b0 00000082 00000002 f4177e24 f4177e18 00000000 00000000 c045f0c0 
       c0461d00 c0461d00 c0461d00 f4177e28 f7a6a204 c3734d00 f756ca80 f7855a5c 
       ffffc6d5 00000000 00000001 f7855a4c ffffffff 00000000 00000000 00000000 
Call Trace:
 [<c0317dc6>] schedule_timeout+0x69/0xa1
 [<c02bf79f>] __skb_recv_datagram+0x65/0x22d
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c02bf989>] skb_recv_datagram+0x22/0x27
 [<c030c4c1>] unix_accept+0x56/0xe3
 [<c02b8b03>] do_accept+0xf0/0x1d4
 [<c02bce78>] skb_dequeue+0x50/0x56
 [<c0181530>] dput+0x97/0x145
 [<c02b8e25>] sys_socketcall+0x23e/0x2a9
 [<c016f61c>] filp_close+0x3e/0x62
 [<c01f4a80>] trace_hardirqs_on_thunk+0xc/0x10
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
php5-cgi      S c01512af     0  4034   3968
       f79e5190 00000082 c342544c c01512af 00000246 c03ed880 0000000c c045f0c0 
       c0461d00 c0461d00 c0461d00 f79e5190 f79e52e4 c3734d00 f756c080 f782ba5c 
       00000046 f36f90ac 00000001 f782ba4c 00000046 00000000 00000002 00000001 
Call Trace:
 [<c01512af>] rmqueue_bulk+0x67/0x71
 [<c0317dc6>] schedule_timeout+0x69/0xa1
 [<c02bf79f>] __skb_recv_datagram+0x65/0x22d
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c02bf989>] skb_recv_datagram+0x22/0x27
 [<c030c4c1>] unix_accept+0x56/0xe3
 [<c02b8b03>] do_accept+0xf0/0x1d4
 [<c015bb55>] handle_mm_fault+0x176/0x8a4
 [<c0134eca>] up_read+0x14/0x2a
 [<c011377d>] do_page_fault+0xa5/0x848
 [<c011382e>] do_page_fault+0x156/0x848
 [<c02b8e25>] sys_socketcall+0x23e/0x2a9
 [<c01f4a80>] trace_hardirqs_on_thunk+0xc/0x10
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
php5-cgi      S f2da3e24     0  4035   4017
       f79e4920 00000082 00000002 f2da3e24 f2da3e18 00000000 00000000 c045f0c0 
       c0461d00 c0461d00 c0461d00 f2da3e28 f79e4a74 c3734d00 f756cd00 f7855a5c 
       ffffe979 00000000 00000001 f7855a4c ffffffff 00000000 00000000 00000000 
Call Trace:
 [<c0317dc6>] schedule_timeout+0x69/0xa1
 [<c02bf79f>] __skb_recv_datagram+0x65/0x22d
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c02bf989>] skb_recv_datagram+0x22/0x27
 [<c030c4c1>] unix_accept+0x56/0xe3
 [<c02b8b03>] do_accept+0xf0/0x1d4
 [<c02bce78>] skb_dequeue+0x50/0x56
 [<c0181530>] dput+0x97/0x145
 [<c02b8e25>] sys_socketcall+0x23e/0x2a9
 [<c016f61c>] filp_close+0x3e/0x62
 [<c01f4a80>] trace_hardirqs_on_thunk+0xc/0x10
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
php5-cgi      S c34260ec     0  4036   4005
       f7a97390 00000082 00000000 c34260ec 00000246 c03ed880 0000000c c045f0c0 
       c0461d00 c0461d00 c0461d00 f7a97390 f7a974e4 c373ed00 f7982080 f759ea5c 
       00000046 f4ca90ac 00000001 f759ea4c 00000046 00000000 00000002 00000001 
Call Trace:
 [<c0317dc6>] schedule_timeout+0x69/0xa1
 [<c02bf79f>] __skb_recv_datagram+0x65/0x22d
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c02bf989>] skb_recv_datagram+0x22/0x27
 [<c030c4c1>] unix_accept+0x56/0xe3
 [<c02b8b03>] do_accept+0xf0/0x1d4
 [<c015bb55>] handle_mm_fault+0x176/0x8a4
 [<c0134eca>] up_read+0x14/0x2a
 [<c011377d>] do_page_fault+0xa5/0x848
 [<c011382e>] do_page_fault+0x156/0x848
 [<c02b8e25>] sys_socketcall+0x23e/0x2a9
 [<c01f4a80>] trace_hardirqs_on_thunk+0xc/0x10
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
php5-cgi      S c3731aa0     0  4037   3980
       f7ab2430 00000082 00000400 c3731aa0 00000002 f4007e20 000800d0 c045f0c0 
       c0461d00 c0461d00 c0461d00 f4007e28 f7ab2584 c3734d00 f6b40580 f783ed5c 
       ffff9d3f 00000001 00000001 f783ed4c ffffffff 00000000 00000000 00000000 
Call Trace:
 [<c0317dc6>] schedule_timeout+0x69/0xa1
 [<c02bf79f>] __skb_recv_datagram+0x65/0x22d
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c02bf989>] skb_recv_datagram+0x22/0x27
 [<c030c4c1>] unix_accept+0x56/0xe3
 [<c02b8b03>] do_accept+0xf0/0x1d4
 [<c015bb55>] handle_mm_fault+0x176/0x8a4
 [<c0134eca>] up_read+0x14/0x2a
 [<c011377d>] do_page_fault+0xa5/0x848
 [<c011382e>] do_page_fault+0x156/0x848
 [<c02b8e25>] sys_socketcall+0x23e/0x2a9
 [<c01f4a80>] trace_hardirqs_on_thunk+0xc/0x10
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
php5-cgi      S 001200d2     0  4038   3980
       f79fc230 00000082 00000000 001200d2 00000246 c03ed880 0000000c c045f0c0 
       c0461d00 c0461d00 c0461d00 f79fc230 f79fc384 c3734d00 f7982300 f783ed5c 
       00000046 f240dd2c 00000001 f783ed4c 00000046 00000000 00000002 00000001 
Call Trace:
 [<c0317dc6>] schedule_timeout+0x69/0xa1
 [<c02bf79f>] __skb_recv_datagram+0x65/0x22d
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c02bf989>] skb_recv_datagram+0x22/0x27
 [<c030c4c1>] unix_accept+0x56/0xe3
 [<c02b8b03>] do_accept+0xf0/0x1d4
 [<c015bb55>] handle_mm_fault+0x176/0x8a4
 [<c0134eca>] up_read+0x14/0x2a
 [<c011377d>] do_page_fault+0xa5/0x848
 [<c011382e>] do_page_fault+0x156/0x848
 [<c02b8e25>] sys_socketcall+0x23e/0x2a9
 [<c01f4a80>] trace_hardirqs_on_thunk+0xc/0x10
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
php5-cgi      S c01512af     0  4039   3980
       f7a0b410 00000082 c34273c8 c01512af 00000246 c03ed880 0000000c c045f0c0 
       c0461d00 c0461d00 c0461d00 f7a0b410 f7a0b564 c373ed00 f64a8800 f783ed5c 
       00000046 f240dd2c 00000001 f783ed4c 00000046 00000000 00000002 00000001 
Call Trace:
 [<c01512af>] rmqueue_bulk+0x67/0x71
 [<c0317dc6>] schedule_timeout+0x69/0xa1
 [<c02bf79f>] __skb_recv_datagram+0x65/0x22d
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c02bf989>] skb_recv_datagram+0x22/0x27
 [<c030c4c1>] unix_accept+0x56/0xe3
 [<c02b8b03>] do_accept+0xf0/0x1d4
 [<c015bb55>] handle_mm_fault+0x176/0x8a4
 [<c0134eca>] up_read+0x14/0x2a
 [<c011377d>] do_page_fault+0xa5/0x848
 [<c011382e>] do_page_fault+0x156/0x848
 [<c02b8e25>] sys_socketcall+0x23e/0x2a9
 [<c01f4a80>] trace_hardirqs_on_thunk+0xc/0x10
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
php5-cgi      S c01512af     0  4040   4005
       f78e4e20 00000082 c3427a14 c01512af 00000246 c03ed880 0000000c c045f0c0 
       c0461d00 c0461d00 c0461d00 f78e4e20 f78e4f74 c3734d00 f6a8d580 f759ea5c 
       00000046 f4ca90ac 00000001 f759ea4c 00000046 00000000 00000002 00000001 
Call Trace:
 [<c01512af>] rmqueue_bulk+0x67/0x71
 [<c0317dc6>] schedule_timeout+0x69/0xa1
 [<c02bf79f>] __skb_recv_datagram+0x65/0x22d
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c02bf989>] skb_recv_datagram+0x22/0x27
 [<c030c4c1>] unix_accept+0x56/0xe3
 [<c02b8b03>] do_accept+0xf0/0x1d4
 [<c015bb55>] handle_mm_fault+0x176/0x8a4
 [<c0134eca>] up_read+0x14/0x2a
 [<c011377d>] do_page_fault+0xa5/0x848
 [<c011382e>] do_page_fault+0x156/0x848
 [<c02b8e25>] sys_socketcall+0x23e/0x2a9
 [<c01f4a80>] trace_hardirqs_on_thunk+0xc/0x10
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
php5-cgi      S 001200d2     0  4041   3980
       f71c9290 00000082 00000000 001200d2 00000246 c03ed880 0000000c c045f0c0 
       c0461d00 c0461d00 c0461d00 f71c9290 f71c93e4 c373ed00 f7967a80 f783ed5c 
       00000046 f240dd2c 00000001 f783ed4c 00000046 00000000 00000002 00000001 
Call Trace:
 [<c0317dc6>] schedule_timeout+0x69/0xa1
 [<c02bf79f>] __skb_recv_datagram+0x65/0x22d
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c02bf989>] skb_recv_datagram+0x22/0x27
 [<c030c4c1>] unix_accept+0x56/0xe3
 [<c02b8b03>] do_accept+0xf0/0x1d4
 [<c015bb55>] handle_mm_fault+0x176/0x8a4
 [<c0134eca>] up_read+0x14/0x2a
 [<c011377d>] do_page_fault+0xa5/0x848
 [<c011382e>] do_page_fault+0x156/0x848
 [<c02b8e25>] sys_socketcall+0x23e/0x2a9
 [<c01f4a80>] trace_hardirqs_on_thunk+0xc/0x10
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
php5-cgi      S 001200d2     0  4042   4005
       f79cb690 00000082 00000000 001200d2 00000246 c03ed880 0000000c c045f0c0 
       c0461d00 c0461d00 c0461d00 f79cb690 f79cb7e4 c3734d00 f6a8da80 f759ea5c 
       00000046 f4ca90ac 00000001 f759ea4c 00000046 00000000 00000002 00000001 
Call Trace:
 [<c0317dc6>] schedule_timeout+0x69/0xa1
 [<c02bf79f>] __skb_recv_datagram+0x65/0x22d
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c02bf989>] skb_recv_datagram+0x22/0x27
 [<c030c4c1>] unix_accept+0x56/0xe3
 [<c02b8b03>] do_accept+0xf0/0x1d4
 [<c015bb55>] handle_mm_fault+0x176/0x8a4
 [<c0134eca>] up_read+0x14/0x2a
 [<c011377d>] do_page_fault+0xa5/0x848
 [<c011382e>] do_page_fault+0x156/0x848
 [<c02b8e25>] sys_socketcall+0x23e/0x2a9
 [<c01f4a80>] trace_hardirqs_on_thunk+0xc/0x10
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
php5-cgi      S f29b9e24     0  4043   3992
       f2d7b210 00000082 00000002 f29b9e24 f29b9e18 00000000 00000000 c045f0c0 
       c0461d00 c0461d00 c0461d00 f29b9e28 f2d7b364 c3734d00 f6a8d080 f75ded5c 
       ffffe350 00000000 00000001 f75ded4c ffffffff 00000000 00000000 00000000 
Call Trace:
 [<c0317dc6>] schedule_timeout+0x69/0xa1
 [<c02bf79f>] __skb_recv_datagram+0x65/0x22d
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c02bf989>] skb_recv_datagram+0x22/0x27
 [<c030c4c1>] unix_accept+0x56/0xe3
 [<c02b8b03>] do_accept+0xf0/0x1d4
 [<c02bce78>] skb_dequeue+0x50/0x56
 [<c0181530>] dput+0x97/0x145
 [<c02b8e25>] sys_socketcall+0x23e/0x2a9
 [<c016f61c>] filp_close+0x3e/0x62
 [<c01f4a80>] trace_hardirqs_on_thunk+0xc/0x10
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
php5-cgi      S 001200d2     0  4044   3992
       f2d7a9a0 00000082 00000000 001200d2 00000246 c03ed880 0000000c c045f0c0 
       c0461d00 c0461d00 c0461d00 f2d7a9a0 f2d7aaf4 c3734d00 f6a8d300 f75ded5c 
       00000046 f240daac 00000001 f75ded4c 00000046 00000000 00000002 00000001 
Call Trace:
 [<c0317dc6>] schedule_timeout+0x69/0xa1
 [<c02bf79f>] __skb_recv_datagram+0x65/0x22d
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c02bf989>] skb_recv_datagram+0x22/0x27
 [<c030c4c1>] unix_accept+0x56/0xe3
 [<c02b8b03>] do_accept+0xf0/0x1d4
 [<c015bb55>] handle_mm_fault+0x176/0x8a4
 [<c0134eca>] up_read+0x14/0x2a
 [<c011377d>] do_page_fault+0xa5/0x848
 [<c011382e>] do_page_fault+0x156/0x848
 [<c02b8e25>] sys_socketcall+0x23e/0x2a9
 [<c01f4a80>] trace_hardirqs_on_thunk+0xc/0x10
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
php5-cgi      S f5009e24     0  4045   3992
       f2d7a130 00000082 00000002 f5009e24 f5009e18 00000000 0000000c c045f0c0 
       c0461d00 c0461d00 c0461d00 f5009e28 f2d7a284 c373ed00 f28a9d00 f75ded5c 
       ffff9d5f 00000000 00000001 f75ded4c ffffffff 00000000 00000000 00000000 
Call Trace:
 [<c0317dc6>] schedule_timeout+0x69/0xa1
 [<c02bf79f>] __skb_recv_datagram+0x65/0x22d
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c02bf989>] skb_recv_datagram+0x22/0x27
 [<c030c4c1>] unix_accept+0x56/0xe3
 [<c02b8b03>] do_accept+0xf0/0x1d4
 [<c015bb55>] handle_mm_fault+0x176/0x8a4
 [<c0134eca>] up_read+0x14/0x2a
 [<c011377d>] do_page_fault+0xa5/0x848
 [<c011382e>] do_page_fault+0x156/0x848
 [<c02b8e25>] sys_socketcall+0x23e/0x2a9
 [<c01f4a80>] trace_hardirqs_on_thunk+0xc/0x10
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
php5-cgi      S c01512af     0  4046   3992
       f29e1290 00000082 c34112c0 c01512af 00000246 c03ed880 0000000c c045f0c0 
       c0461d00 c0461d00 c0461d00 f29e1290 f29e13e4 c3734d00 f28a9a80 f75ded5c 
       00000046 f240daac 00000001 f75ded4c 00000046 00000000 00000002 00000001 
Call Trace:
 [<c01512af>] rmqueue_bulk+0x67/0x71
 [<c0317dc6>] schedule_timeout+0x69/0xa1
 [<c02bf79f>] __skb_recv_datagram+0x65/0x22d
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c02bf989>] skb_recv_datagram+0x22/0x27
 [<c030c4c1>] unix_accept+0x56/0xe3
 [<c02b8b03>] do_accept+0xf0/0x1d4
 [<c015bb55>] handle_mm_fault+0x176/0x8a4
 [<c0134eca>] up_read+0x14/0x2a
 [<c011377d>] do_page_fault+0xa5/0x848
 [<c011382e>] do_page_fault+0x156/0x848
 [<c02b8e25>] sys_socketcall+0x23e/0x2a9
 [<c01f4a80>] trace_hardirqs_on_thunk+0xc/0x10
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
php5-cgi      S f29e5e24     0  4047   3992
       f29e0a20 00000082 00000002 f29e5e24 f29e5e18 00000000 0000000c c045f0c0 
       c0461d00 c0461d00 c0461d00 f29e5e28 f29e0b74 c373ed00 f28a9800 f75ded5c 
       ffff9d60 00000000 00000001 f75ded4c ffffffff 00000000 00000000 00000000 
Call Trace:
 [<c0317dc6>] schedule_timeout+0x69/0xa1
 [<c02bf79f>] __skb_recv_datagram+0x65/0x22d
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c02bf989>] skb_recv_datagram+0x22/0x27
 [<c030c4c1>] unix_accept+0x56/0xe3
 [<c02b8b03>] do_accept+0xf0/0x1d4
 [<c015bb55>] handle_mm_fault+0x176/0x8a4
 [<c0134eca>] up_read+0x14/0x2a
 [<c011377d>] do_page_fault+0xa5/0x848
 [<c011382e>] do_page_fault+0x156/0x848
 [<c02b8e25>] sys_socketcall+0x23e/0x2a9
 [<c01f4a80>] trace_hardirqs_on_thunk+0xc/0x10
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
php5-cgi      S f7054bb4     0  4048   3992
       f29e01b0 00000082 00000000 f7054bb4 f29e01b0 f29e01b0 000800d0 c045f0c0 
       c0461d00 c0461d00 c0461d00 f29e01b0 f29e0304 c3734d00 f28a9580 f75ded5c 
       00000046 f240daac 00000001 f75ded4c 00000046 00000000 00000002 00000001 
Call Trace:
 [<c0317dc6>] schedule_timeout+0x69/0xa1
 [<c02bf79f>] __skb_recv_datagram+0x65/0x22d
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c02bf989>] skb_recv_datagram+0x22/0x27
 [<c030c4c1>] unix_accept+0x56/0xe3
 [<c02b8b03>] do_accept+0xf0/0x1d4
 [<c015bb55>] handle_mm_fault+0x176/0x8a4
 [<c0134eca>] up_read+0x14/0x2a
 [<c011377d>] do_page_fault+0xa5/0x848
 [<c011382e>] do_page_fault+0x156/0x848
 [<c02b8e25>] sys_socketcall+0x23e/0x2a9
 [<c01f4a80>] trace_hardirqs_on_thunk+0xc/0x10
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
php5-cgi      S f29c5e24     0  4049   3992
       f29c3310 00000082 00000002 f29c5e24 f29c5e18 00000000 0000000c c045f0c0 
       c0461d00 c0461d00 c0461d00 f29c5e28 f29c3464 c373ed00 f28a9300 f75ded5c 
       ffff9d60 00000000 00000001 f75ded4c ffffffff 00000000 00000000 00000000 
Call Trace:
 [<c0317dc6>] schedule_timeout+0x69/0xa1
 [<c02bf79f>] __skb_recv_datagram+0x65/0x22d
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c02bf989>] skb_recv_datagram+0x22/0x27
 [<c030c4c1>] unix_accept+0x56/0xe3
 [<c02b8b03>] do_accept+0xf0/0x1d4
 [<c015bb55>] handle_mm_fault+0x176/0x8a4
 [<c0134eca>] up_read+0x14/0x2a
 [<c011377d>] do_page_fault+0xa5/0x848
 [<c011382e>] do_page_fault+0x156/0x848
 [<c02b8e25>] sys_socketcall+0x23e/0x2a9
 [<c01f4a80>] trace_hardirqs_on_thunk+0xc/0x10
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
php5-cgi      S 001200d2     0  4050   3992
       f29c2aa0 00000082 00000000 001200d2 00000246 c03ed880 0000000c c045f0c0 
       c0461d00 c0461d00 c0461d00 f29c2aa0 f29c2bf4 c3734d00 f28a9080 f75ded5c 
       00000046 f240daac 00000001 f75ded4c 00000046 00000000 00000002 00000001 
Call Trace:
 [<c0317dc6>] schedule_timeout+0x69/0xa1
 [<c02bf79f>] __skb_recv_datagram+0x65/0x22d
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c02bf989>] skb_recv_datagram+0x22/0x27
 [<c030c4c1>] unix_accept+0x56/0xe3
 [<c02b8b03>] do_accept+0xf0/0x1d4
 [<c015bb55>] handle_mm_fault+0x176/0x8a4
 [<c0134eca>] up_read+0x14/0x2a
 [<c011377d>] do_page_fault+0xa5/0x848
 [<c011382e>] do_page_fault+0x156/0x848
 [<c02b8e25>] sys_socketcall+0x23e/0x2a9
 [<c01f4a80>] trace_hardirqs_on_thunk+0xc/0x10
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
bash          S 00000000     0  4055   2969
       f6a68630 00000082 00000002 00000000 f7191c3c f6a68630 c03eb0b0 c045f0c0 
       c0461d00 c0461d00 c0461d00 00000000 f6a68784 c3734d00 f7801580 00000000 
       00000002 00000001 00000046 f719197c f7191800 f7192000 00000046 00000000 
Call Trace:
 [<c0317dc6>] schedule_timeout+0x69/0xa1
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c022ff41>] read_chan+0x1e1/0x640
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c022c03a>] tty_read+0x70/0x9d
 [<c022fd60>] read_chan+0x0/0x640
 [<c022bfca>] tty_read+0x0/0x9d
 [<c01715f8>] vfs_read+0x85/0x11b
 [<c017192b>] sys_read+0x41/0x6a
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
bash          S f1857eb4     0  4078   2965
       f2836530 00000082 00000002 f1857eb4 f1857ea8 00000000 c03eb0b0 c045f0c0 
       c0461d00 c0461d00 c0461d00 f1857eb8 f2836684 c3734d00 f7832300 00000000 
       ffffe335 00000000 00000046 f748817c ffffffff f651a000 00000000 00000000 
Call Trace:
 [<c0317dc6>] schedule_timeout+0x69/0xa1
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c022ff41>] read_chan+0x1e1/0x640
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c022c03a>] tty_read+0x70/0x9d
 [<c022fd60>] read_chan+0x0/0x640
 [<c022bfca>] tty_read+0x0/0x9d
 [<c01715f8>] vfs_read+0x85/0x11b
 [<c017192b>] sys_read+0x41/0x6a
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
mailmanctl    S f2017f24     0  4100      1
       f2837610 00200082 00000002 f2017f24 f2017f18 00000000 f2837610 c045f0c0 
       c0461d00 c0461d00 c0461d00 f2017f28 f2837764 c373ed00 f7801080 00000000 
       ffffedab 00000000 00000000 00000000 ffffffff 00200246 00000000 00000000 
Call Trace:
 [<c0122ad7>] do_wait+0x28c/0x342
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c0122bf6>] sys_wait4+0x69/0xa4
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
python        R running      0  4101   4100
       f29c2230 00000086 00000002 f18bbb14 f18bbb08 00000000 c04b9e00 c045f0c0 
       c0461d00 c0461d00 c0461d00 f18bbb18 f29c2384 c3734d00 f7967080 c04b9e00 
       ffffef43 00000000 00000286 00000286 ffffffff c04b9e00 00000000 00000000 
Call Trace:
 [<c0317da1>] schedule_timeout+0x44/0xa1
 [<c01287ec>] process_timeout+0x0/0x5
 [<c017cbae>] do_select+0x3ef/0x4cb
 [<c017c7d3>] do_select+0x14/0x4cb
 [<c017cfbf>] __pollwait+0x0/0xba
 [<c014bd71>] cpupri_set+0x96/0xbc
 [<c011a4c9>] __enqueue_rt_entity+0x87/0x151
 [<c011abae>] try_to_wake_up+0xa8/0x137
 [<c0117dce>] __wake_up_common+0x46/0x68
 [<c0319826>] _read_unlock+0x14/0x1c
 [<c02ba2fd>] sock_queue_rcv_skb+0x8d/0xc6
 [<c0319867>] _spin_unlock+0x14/0x1c
 [<c02f86dd>] udp_queue_rcv_skb+0x11b/0x223
 [<c02f7f61>] __udp4_lib_lookup+0xed/0x110
 [<c02f8b09>] __udp4_lib_rcv+0x324/0x7a6
 [<f8972087>] ipt_local_in_hook+0x0/0x19 [iptable_filter]
 [<c02db3f7>] ip_local_deliver_finish+0x10d/0x15f
 [<c02db329>] ip_local_deliver_finish+0x3f/0x15f
 [<c0108bbd>] nommu_map_single+0x0/0x63
 [<c017ce7d>] core_sys_select+0x1f3/0x335
 [<c017ccac>] core_sys_select+0x22/0x335
 [<f89483e7>] e1000_clean_rx_irq+0x301/0x41a [e1000]
 [<f89480e6>] e1000_clean_rx_irq+0x0/0x41a [e1000]
 [<f89459a9>] e1000_clean+0x450/0x5f9 [e1000]
 [<f8a4ef2e>] gfs_readdir+0x329/0x3c2 [gfs]
 [<f8a4d530>] filldir_reg_func+0x0/0x1c9 [gfs]
 [<c01f5002>] copy_to_user+0x2f/0x4d
 [<c017beec>] filldir64+0x0/0xc5
 [<c02c4724>] net_rx_action+0x85/0x1db
 [<c02c473c>] net_rx_action+0x9d/0x1db
 [<c017d14f>] sys_select+0xd6/0x19e
 [<c0124699>] _local_bh_enable+0x44/0xa0
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
python        R running      0  4102   4100
       f203b310 00000086 00000046 00000000 00000002 00000001 c04b9e00 c045f0c0 
       c0461d00 c0461d00 c0461d00 00000000 f203b464 c3734d00 f78a5d00 c04b9e00 
       00000286 c04b9e00 00000286 00000286 f20a9b38 c04b9e00 c0128a5a 00000000 
Call Trace:
 [<c0128a5a>] __mod_timer+0x94/0xa3
 [<c0317da1>] schedule_timeout+0x44/0xa1
 [<c01287ec>] process_timeout+0x0/0x5
 [<c017cbae>] do_select+0x3ef/0x4cb
 [<c017c7d3>] do_select+0x14/0x4cb
 [<c017cfbf>] __pollwait+0x0/0xba
 [<c0317428>] wait_for_common+0x26/0x11b
 [<c014c8aa>] find_get_page+0xa2/0xbf
 [<c014c8aa>] find_get_page+0xa2/0xbf
 [<c014c815>] find_get_page+0xd/0xbf
 [<f8a31645>] do_filldir_main+0x28/0x1d1 [gfs]
 [<f8a338e3>] compare_dents+0x0/0x65 [gfs]
 [<c0317428>] wait_for_common+0x26/0x11b
 [<c017ce7d>] core_sys_select+0x1f3/0x335
 [<c017ccac>] core_sys_select+0x22/0x335
 [<f8a3a8cd>] gfs_glock_dq+0xf0/0x157 [gfs]
 [<c031995f>] _spin_lock+0x31/0x3c
 [<f8a3a877>] gfs_glock_dq+0x9a/0x157 [gfs]
 [<c017beec>] filldir64+0x0/0xc5
 [<c0319867>] _spin_unlock+0x14/0x1c
 [<f8a4ef2e>] gfs_readdir+0x329/0x3c2 [gfs]
 [<f8a4d530>] filldir_reg_func+0x0/0x1c9 [gfs]
 [<c01f5002>] copy_to_user+0x2f/0x4d
 [<c017beec>] filldir64+0x0/0xc5
 [<c0171bb2>] file_kill+0x13/0x2d
 [<c017d14f>] sys_select+0xd6/0x19e
 [<c01f5002>] copy_to_user+0x2f/0x4d
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
python        R running      0  4103   4100
       f203aaa0 00000086 00000000 c3726014 f203aaa0 f18c5ea0 00000246 c045f0c0 
       c0461d00 c0461d00 c0461d00 00000002 f203abf4 c3734d00 f7801800 c02a6018 
       00000246 00000246 f75a0c90 f75a0c90 f18c5e9c c3726004 c0319826 f64f4680 
Call Trace:
 [<c02a6018>] dm_get_table+0x11/0x2e
 [<c0319826>] _read_unlock+0x14/0x1c
 [<c02a71b0>] dm_table_unplug_all+0x26/0x35
 [<c0317d1b>] io_schedule+0x1b/0x24
 [<c014ca1b>] sync_page+0x2c/0x39
 [<c0317f60>] __wait_on_bit+0x42/0x5e
 [<c014c9ef>] sync_page+0x0/0x39
 [<c014cc20>] wait_on_page_bit+0x64/0x6b
 [<c0131663>] wake_bit_function+0x0/0x3c
 [<c014d11c>] wait_on_page_writeback_range+0x9c/0x104
 [<f8a54178>] gfs_sync_page_i+0x28/0x5c [gfs]
 [<f8a537a9>] gfs_drop_inode+0x55/0x5d [gfs]
 [<c018253a>] iput+0x44/0x4a
 [<c017a157>] do_unlinkat+0xc6/0x140
 [<c015f54c>] do_munmap+0x183/0x1e0
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
python        R running      0  4104   4100
       00000283 00000283 000000e0 f71a0000 f88ed512 0000002a 00000001 c04b9e00 
       00000286 f18c9b38 f18c9b18 c0319cc1 f71a0618 0000002a f7874f00 00000046 
       c04b9e00 00000286 c04b9e00 f71a0000 f7097080 00009b38 00df0000 f71a0580 
Call Trace:
 [<f88ed512>] e1000_xmit_frame+0x7a5/0xb7b [e1000e]
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c0108c32>] read_tsc+0x6/0x28
 [<c01182b1>] sched_slice+0x31/0x37
 [<c0108c32>] read_tsc+0x6/0x28
 [<c0128526>] run_timer_softirq+0x30/0x192
 [<c0319c5a>] _spin_unlock_irq+0x20/0x23
 [<c0128660>] run_timer_softirq+0x16a/0x192
 [<c0124797>] __do_softirq+0xa2/0xf9
 [<c0124699>] _local_bh_enable+0x44/0xa0
 [<c01f4a80>] trace_hardirqs_on_thunk+0xc/0x10
 [<c0103aa8>] restore_nocheck_notrace+0x0/0xe
 [<c0109398>] native_read_tsc+0x8/0xf
 [<c01f4759>] delay_tsc+0x12/0x47
 [<c01f4716>] __delay+0x6/0x7
 [<c01f82ae>] _raw_spin_lock+0xe6/0x170
 [<c018247f>] igrab+0xd/0x3b
 [<c018247f>] igrab+0xd/0x3b
 [<f8a3f161>] gfs_iget+0x38/0x1e7 [gfs]
 [<f8a3f3ad>] gfs_inode_attr_in+0xb/0x26 [gfs]
 [<f8a3c502>] inode_go_unlock+0x20/0x2a [gfs]
 [<f8a3a8a2>] gfs_glock_dq+0xc5/0x157 [gfs]
 [<f8a3ab65>] gfs_glock_dq_uninit+0x8/0x10 [gfs]
 [<f8a51c90>] gfs_permission+0x59/0x63 [gfs]
 [<c017786b>] inode_permission+0x57/0x95
 [<c017902f>] __link_path_walk+0x69/0xbf0
 [<c017ccac>] core_sys_select+0x22/0x335
 [<c0179bed>] path_walk+0x37/0x73
 [<c0179cc0>] do_path_lookup+0x63/0x10d
 [<c0178efb>] getname+0x89/0x98
 [<c017a5fc>] user_path_at+0x37/0x68
 [<c01f5002>] copy_to_user+0x2f/0x4d
 [<c017a66a>] sys_linkat+0x3d/0xf5
 [<c01f5002>] copy_to_user+0x2f/0x4d
 [<c017d1fc>] sys_select+0x183/0x19e
 [<c017a751>] sys_link+0x2f/0x33
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
python        R running      0  4105   4100
       f18cb390 00000086 00000002 f18cdb14 f18cdb08 00000000 f706e000 c045f0c0 
       c0461d00 c0461d00 c0461d00 f18cdb18 f18cb4e4 c373ed00 f7c60800 f706e000 
       ffffef7b 00000000 00000286 00000286 ffffffff f706e000 00000000 00000000 
Call Trace:
 [<c0317da1>] schedule_timeout+0x44/0xa1
 [<c01287ec>] process_timeout+0x0/0x5
 [<c017cbae>] do_select+0x3ef/0x4cb
 [<c017c7d3>] do_select+0x14/0x4cb
 [<c017cfbf>] __pollwait+0x0/0xba
 [<c0317428>] wait_for_common+0x26/0x11b
 [<c0317428>] wait_for_common+0x26/0x11b
 [<c0319c5a>] _spin_unlock_irq+0x20/0x23
 [<c03174cf>] wait_for_common+0xcd/0x11b
 [<c014c8aa>] find_get_page+0xa2/0xbf
 [<f8a5df2f>] gfs_sort+0x80/0x103 [gfs]
 [<f8a31645>] do_filldir_main+0x28/0x1d1 [gfs]
 [<f8a338e3>] compare_dents+0x0/0x65 [gfs]
 [<f8a2ecd4>] getbuf+0xfe/0x196 [gfs]
 [<c0317428>] wait_for_common+0x26/0x11b
 [<c017ce7d>] core_sys_select+0x1f3/0x335
 [<c017ccac>] core_sys_select+0x22/0x335
 [<f8a3a8cd>] gfs_glock_dq+0xf0/0x157 [gfs]
 [<c031995f>] _spin_lock+0x31/0x3c
 [<f8a3a877>] gfs_glock_dq+0x9a/0x157 [gfs]
 [<c017beec>] filldir64+0x0/0xc5
 [<c0319867>] _spin_unlock+0x14/0x1c
 [<f8a4ef2e>] gfs_readdir+0x329/0x3c2 [gfs]
 [<f8a4d530>] filldir_reg_func+0x0/0x1c9 [gfs]
 [<c01f5002>] copy_to_user+0x2f/0x4d
 [<c017beec>] filldir64+0x0/0xc5
 [<c0181542>] dput+0xa9/0x145
 [<c017d14f>] sys_select+0xd6/0x19e
 [<c016f61c>] filp_close+0x3e/0x62
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
python        R running      0  4106   4100
       f18cab20 00000086 00000046 00000000 00000002 00000001 c04b9e00 c045f0c0 
       c0461d00 c0461d00 c0461d00 00000000 f18cac74 c3734d00 f7c60580 c04b9e00 
       00000286 c04b9e00 00000286 00000286 f18cfb38 c04b9e00 c0128a5a 00000000 
Call Trace:
 [<c0128a5a>] __mod_timer+0x94/0xa3
 [<c0317da1>] schedule_timeout+0x44/0xa1
 [<c01287ec>] process_timeout+0x0/0x5
 [<c017cbae>] do_select+0x3ef/0x4cb
 [<c017c7d3>] do_select+0x14/0x4cb
 [<f89459a9>] e1000_clean+0x450/0x5f9 [e1000]
 [<c014bd71>] cpupri_set+0x96/0xbc
 [<c017cfbf>] __pollwait+0x0/0xba
 [<c02c4724>] net_rx_action+0x85/0x1db
 [<c02c473c>] net_rx_action+0x9d/0x1db
 [<c0124797>] __do_softirq+0xa2/0xf9
 [<c0124699>] _local_bh_enable+0x44/0xa0
 [<c014c8aa>] find_get_page+0xa2/0xbf
 [<c014c8aa>] find_get_page+0xa2/0xbf
 [<c014c815>] find_get_page+0xd/0xbf
 [<f8a31645>] do_filldir_main+0x28/0x1d1 [gfs]
 [<f8a338e3>] compare_dents+0x0/0x65 [gfs]
 [<c0317428>] wait_for_common+0x26/0x11b
 [<c017ce7d>] core_sys_select+0x1f3/0x335
 [<c017ccac>] core_sys_select+0x22/0x335
 [<f8a3a8cd>] gfs_glock_dq+0xf0/0x157 [gfs]
 [<c031995f>] _spin_lock+0x31/0x3c
 [<f8a3a877>] gfs_glock_dq+0x9a/0x157 [gfs]
 [<c017beec>] filldir64+0x0/0xc5
 [<c0319867>] _spin_unlock+0x14/0x1c
 [<f8a4ef2e>] gfs_readdir+0x329/0x3c2 [gfs]
 [<f8a4d530>] filldir_reg_func+0x0/0x1c9 [gfs]
 [<c01f5002>] copy_to_user+0x2f/0x4d
 [<c017beec>] filldir64+0x0/0xc5
 [<c0171bb2>] file_kill+0x13/0x2d
 [<c017d14f>] sys_select+0xd6/0x19e
 [<c01f5002>] copy_to_user+0x2f/0x4d
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
python        R running      0  4107   4100
       f18ca2b0 00000086 00000046 00000000 00000002 00000001 c04b9e00 c045f0c0 
       c0461d00 c0461d00 c0461d00 00000000 f18ca404 c3734d00 f7c60300 c04b9e00 
       00000286 c04b9e00 00000286 00000286 f5d0fb38 c04b9e00 c0128a5a 00000000 
Call Trace:
 [<c0128a5a>] __mod_timer+0x94/0xa3
 [<c0317da1>] schedule_timeout+0x44/0xa1
 [<c01287ec>] process_timeout+0x0/0x5
 [<c017cbae>] do_select+0x3ef/0x4cb
 [<c017c7d3>] do_select+0x14/0x4cb
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c017cfbf>] __pollwait+0x0/0xba
 [<c0317428>] wait_for_common+0x26/0x11b
 [<c0317428>] wait_for_common+0x26/0x11b
 [<c0319c5a>] _spin_unlock_irq+0x20/0x23
 [<f8a3a8cd>] gfs_glock_dq+0xf0/0x157 [gfs]
 [<c031995f>] _spin_lock+0x31/0x3c
 [<f8a3a877>] gfs_glock_dq+0x9a/0x157 [gfs]
 [<c0181530>] dput+0x97/0x145
 [<f8a4ce96>] gfs_drevalidate+0x1a6/0x1f0 [gfs]
 [<c017ce7d>] core_sys_select+0x1f3/0x335
 [<c017ccac>] core_sys_select+0x22/0x335
 [<f8a3a8cd>] gfs_glock_dq+0xf0/0x157 [gfs]
 [<c031995f>] _spin_lock+0x31/0x3c
 [<f8a3a877>] gfs_glock_dq+0x9a/0x157 [gfs]
 [<c01f5002>] copy_to_user+0x2f/0x4d
 [<c0173c33>] cp_new_stat64+0xf9/0x10b
 [<c017d14f>] sys_select+0xd6/0x19e
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
python        S f2015b14     0  4108   4100
       f283b410 00000086 00000002 f2015b14 f2015b08 00000000 c04b9e00 c045f0c0 
       c0461d00 c0461d00 c0461d00 f2015b18 f283b564 c3734d00 f64a8080 c04b9e00 
       ffffeddf 00000000 00000286 00000286 ffffffff c04b9e00 00000000 00000000 
Call Trace:
 [<c0317da1>] schedule_timeout+0x44/0xa1
 [<c01287ec>] process_timeout+0x0/0x5
 [<c017cbae>] do_select+0x3ef/0x4cb
 [<c017c7d3>] do_select+0x14/0x4cb
 [<f89459a9>] e1000_clean+0x450/0x5f9 [e1000]
 [<c017cfbf>] __pollwait+0x0/0xba
 [<c0317428>] wait_for_common+0x26/0x11b
 [<c0317428>] wait_for_common+0x26/0x11b
 [<c0319c5a>] _spin_unlock_irq+0x20/0x23
 [<c03174cf>] wait_for_common+0xcd/0x11b
 [<c014c8aa>] find_get_page+0xa2/0xbf
 [<f8a5df2f>] gfs_sort+0x80/0x103 [gfs]
 [<f8a31645>] do_filldir_main+0x28/0x1d1 [gfs]
 [<f8a338e3>] compare_dents+0x0/0x65 [gfs]
 [<f8a2ecd4>] getbuf+0xfe/0x196 [gfs]
 [<c0317428>] wait_for_common+0x26/0x11b
 [<c017ce7d>] core_sys_select+0x1f3/0x335
 [<c017ccac>] core_sys_select+0x22/0x335
 [<f8a3a8cd>] gfs_glock_dq+0xf0/0x157 [gfs]
 [<c031995f>] _spin_lock+0x31/0x3c
 [<f8a3a877>] gfs_glock_dq+0x9a/0x157 [gfs]
 [<c017beec>] filldir64+0x0/0xc5
 [<c0319867>] _spin_unlock+0x14/0x1c
 [<f8a4ef2e>] gfs_readdir+0x329/0x3c2 [gfs]
 [<f8a4d530>] filldir_reg_func+0x0/0x1c9 [gfs]
 [<c017beec>] filldir64+0x0/0xc5
 [<c0181542>] dput+0xa9/0x145
 [<c017d14f>] sys_select+0xd6/0x19e
 [<c016f61c>] filp_close+0x3e/0x62
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
smtpd         R running      0  4109   3362
       f283aba0 00200086 00200046 00000000 00000002 00000001 c04b9e00 c045f0c0 
       c0461d00 c0461d00 c0461d00 00000000 f283acf4 c3734d00 f64a8d00 c04b9e00 
       00200282 c04b9e00 00200282 00200282 f143bf40 c04b9e00 c0128a5a 00000000 
Call Trace:
 [<c0128a5a>] __mod_timer+0x94/0xa3
 [<c0317da1>] schedule_timeout+0x44/0xa1
 [<c01287ec>] process_timeout+0x0/0x5
 [<c0199fe9>] sys_epoll_wait+0x15b/0x500
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
proxymap      S 00000002     0  4110   3362
       f283a330 00200086 c3731f30 00000002 00000000 f1916ed0 f283a330 c045f0c0 
       c0461d00 c0461d00 c0461d00 00200046 f283a484 c3734d00 f7896800 f1916e90 
       f283a330 00000002 00000001 00000000 f1916ea0 00200046 f1916e90 f1916e80 
Call Trace:
 [<c0317dc6>] schedule_timeout+0x69/0xa1
 [<c0199fe9>] sys_epoll_wait+0x15b/0x500
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
trivial-rewri R running      0  4111   3362
       f143f490 00200086 00000002 f4851f1c f4851f10 00000000 c04b9e00 c045f0c0 
       c0461d00 c0461d00 c0461d00 f4851f20 f143f5e4 c3734d00 f6b40d00 c04b9e00 
       ffffef29 00000000 00200282 00200282 ffffffff c04b9e00 00000000 00000000 
Call Trace:
 [<c0317da1>] schedule_timeout+0x44/0xa1
 [<c01287ec>] process_timeout+0x0/0x5
 [<c0199fe9>] sys_epoll_wait+0x15b/0x500
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
cleanup       R running      0  4112   3362
       f2836da0 00200086 00200046 00000000 00000002 00000001 c04b9e00 c045f0c0 
       c0461d00 c0461d00 c0461d00 00000000 f2836ef4 c3734d00 f64a8580 c04b9e00 
       00200282 c04b9e00 00200282 00200282 f180df40 c04b9e00 c0128a5a 00000000 
Call Trace:
 [<c0128a5a>] __mod_timer+0x94/0xa3
 [<c0317da1>] schedule_timeout+0x44/0xa1
 [<c01287ec>] process_timeout+0x0/0x5
 [<c0199fe9>] sys_epoll_wait+0x15b/0x500
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
smtp          S f19d9f50     0  4113   3362
       f21781b0 00200086 00000002 f19d9f50 f19d9f44 00000000 00200046 c045f0c0 
       c0461d00 c0461d00 c0461d00 f19d9f54 f2178304 c373ed00 f2033d00 00200282 
       ffffef4b 00000000 c0319cc1 00000000 ffffffff 00200046 00000000 00000000 
Call Trace:
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c017f0b7>] flock_lock_file_wait+0x177/0x275
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c017f2d2>] sys_flock+0x11d/0x122
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
smtp          S f154ff50     0  4114   3362
       f2179290 00200086 00000002 f154ff50 f154ff44 00000000 00200046 c045f0c0 
       c0461d00 c0461d00 c0461d00 f154ff54 f21793e4 c373ed00 f6b40080 00200282 
       ffffef66 00000000 c0319cc1 00000000 ffffffff 00200046 00000000 00000000 
Call Trace:
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c017f0b7>] flock_lock_file_wait+0x177/0x275
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c017f2d2>] sys_flock+0x11d/0x122
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
smtp          S 00000000     0  4115   3362
       f2178a20 00200086 00200046 00000000 00000002 00000001 00200046 c045f0c0 
       c0461d00 c0461d00 c0461d00 00000000 f2178b74 c3734d00 f7c60080 00200282 
       00000001 f1027f44 c0319cc1 00000000 00200046 00200046 f1027f44 00200282 
Call Trace:
 [<c0319cc1>] _spin_lock_irqsave+0x41/0x49
 [<c017f0b7>] flock_lock_file_wait+0x177/0x275
 [<c013162e>] autoremove_wake_function+0x0/0x35
 [<c017f2d2>] sys_flock+0x11d/0x122
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
smtp          R running      0  4116   3362
       f1dbe67c f1dbf0c0 c0182513 f1dbe674 c0180230 f1dbe674 f1dbe67c c0181511 
       00000008 f1dbf0c0 c0171dc1 00000040 00000000 f1dbe674 f7020680 f2091980 
       f7041500 00000000 f2091980 c016f61c 0000000c f7041500 f7041580 0000000c 
Call Trace:
 [<c0182513>] iput+0x1d/0x4a
 [<c0180230>] d_kill+0x2b/0x44
 [<c0181511>] dput+0x78/0x145
 [<c0171dc1>] __fput+0x127/0x17f
 [<c016f61c>] filp_close+0x3e/0x62
 [<c017077e>] sys_close+0x5b/0x97
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
smtp          S 00000000     0  4117   3362
       f1504ca0 00200086 00200046 00000000 00000002 00000001 c04b9e00 c045f0c0 
       c0461d00 c0461d00 c0461d00 00000000 f1504df4 c3734d00 f7963300 c04b9e00 
       00200282 c04b9e00 00200282 00200282 f154bf40 c04b9e00 c0128a5a 00000000 
Call Trace:
 [<c0128a5a>] __mod_timer+0x94/0xa3
 [<c0317da1>] schedule_timeout+0x44/0xa1
 [<c01287ec>] process_timeout+0x0/0x5
 [<c0199fe9>] sys_epoll_wait+0x15b/0x500
 [<c011ac3d>] default_wake_function+0x0/0x8
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
SysRq : Show backtrace of all active CPUs
CPU0:

Pid: 4116, comm: smtp Not tainted (2.6.27.21 #6)
EIP: 0060:[<c010939e>] EFLAGS: 00200246 CPU: 0
EIP is at native_read_tsc+0xe/0xf
EAX: ad617fe3 EBX: 00000000 ECX: 00000000 EDX: 000000f3
ESI: ad617f17 EDI: 00000001 EBP: 00000000 ESP: f1551eec
 DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
CR0: 8005003b CR2: b7c6a84d CR3: 31011000 CR4: 000006f0
DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
DR6: ffff0ff0 DR7: 00000400
 [<c01f4771>] delay_tsc+0x2a/0x47
 [<c01f4716>] __delay+0x6/0x7
 [<c01f82ae>] _raw_spin_lock+0xe6/0x170
 [<c01eeb99>] _atomic_dec_and_lock+0x31/0x58
 [<c01eeb99>] _atomic_dec_and_lock+0x31/0x58
 [<c0182513>] iput+0x1d/0x4a
 [<c0180230>] d_kill+0x2b/0x44
 [<c0181511>] dput+0x78/0x145
 [<c0171dc1>] __fput+0x127/0x17f
 [<c016f61c>] filp_close+0x3e/0x62
 [<c017077e>] sys_close+0x5b/0x97
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
SysRq : Emergency Sync
SysRq : Emergency Sync
SysRq : Show Regs

Pid: 4116, comm: smtp Not tainted (2.6.27.21 #6)
EIP: 0060:[<c01f4771>] EFLAGS: 00200246 CPU: 0
EIP is at delay_tsc+0x2a/0x47
EAX: b8030ea1 EBX: 00000000 ECX: 00000000 EDX: 0000010b
ESI: b8030dd5 EDI: 00000001 EBP: 00000000 ESP: f1551ef0
 DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
CR0: 8005003b CR2: b7c6a84d CR3: 31011000 CR4: 000006f0
DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
DR6: ffff0ff0 DR7: 00000400
 [<c01f4716>] __delay+0x6/0x7
 [<c01f82ae>] _raw_spin_lock+0xe6/0x170
 [<c01eeb99>] _atomic_dec_and_lock+0x31/0x58
 [<c01eeb99>] _atomic_dec_and_lock+0x31/0x58
 [<c0182513>] iput+0x1d/0x4a
 [<c0180230>] d_kill+0x2b/0x44
 [<c0181511>] dput+0x78/0x145
 [<c0171dc1>] __fput+0x127/0x17f
 [<c016f61c>] filp_close+0x3e/0x62
 [<c017077e>] sys_close+0x5b/0x97
 [<c01039e1>] sysenter_do_call+0x12/0x35
 =======================
SysRq : Show Memory
Mem-Info:
DMA per-cpu:
CPU    0: hi:    0, btch:   1 usd:   0
CPU    1: hi:    0, btch:   1 usd:   0
Normal per-cpu:
CPU    0: hi:  186, btch:  31 usd: 133
CPU    1: hi:  186, btch:  31 usd:  70
HighMem per-cpu:
CPU    0: hi:  186, btch:  31 usd:  10
CPU    1: hi:  186, btch:  31 usd:  40
Active:53773 inactive:23336 dirty:1098 writeback:0 unstable:0
 free:679348 slab:13056 mapped:11237 pagetables:922 bounce:0
DMA free:8804kB min:1140kB low:1424kB high:1708kB active:0kB inactive:0kB present:15788kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 868 3015 3015
Normal free:784680kB min:64392kB low:80488kB high:96588kB active:3792kB inactive:9156kB present:889680kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 17174 17174
HighMem free:1923908kB min:512kB low:40288kB high:80064kB active:211300kB inactive:84188kB present:2198284kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 3*4kB 3*8kB 2*16kB 3*32kB 3*64kB 4*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8804kB
Normal: 104*4kB 367*8kB 229*16kB 82*32kB 63*64kB 29*128kB 23*256kB 7*512kB 8*1024kB 4*2048kB 181*4096kB = 784616kB
HighMem: 1*4kB 0*8kB 1*16kB 2*32kB 1*64kB 1*128kB 0*256kB 1*512kB 2*1024kB 0*2048kB 469*4096kB = 1923860kB
32045 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap  = 979832kB
Total swap = 979832kB
786176 pages RAM
556800 pages HighMem
12334 pages reserved
58472 pages shared
71957 pages non-shared
SysRq : Show Blocked State
  task                PC stack   pid father
SysRq : Resetting

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]