[dm-devel] linux-2.6: LVM deadlocks and oops
Hector Oron
hector.oron at gmail.com
Wed Sep 21 09:52:14 UTC 2011
Hello,
While playing upstream patched kernel for i.MX53 LOCO boards (ARM).
I have hit a deadlock and oops on LVM, so I think it is worth it to
document the problem here and maybe it can get fixed.
I have been able to reproduce, so likely I might be able to build
and test suggested patches.
[74093.782238] kjournald starting. Commit interval 5 seconds
[74093.821968] EXT3-fs (dm-4): using internal journal
[74093.826917] EXT3-fs (dm-4): mounted filesystem with ordered data mode
[74397.131757] kjournald starting. Commit interval 5 seconds
[74397.169996] EXT3-fs (dm-4): using internal journal
[74397.174838] EXT3-fs (dm-4): mounted filesystem with ordered data mode
[76601.907689] kjournald starting. Commit interval 5 seconds
[76601.945944] EXT3-fs (dm-4): using internal journal
[76601.950779] EXT3-fs (dm-4): mounted filesystem with ordered data mode
[76656.111857] kjournald starting. Commit interval 5 seconds
[76656.150092] EXT3-fs (dm-4): using internal journal
[76656.154937] EXT3-fs (dm-4): mounted filesystem with ordered data mode
[76793.872136] kjournald starting. Commit interval 5 seconds
[76793.910358] EXT3-fs (dm-4): using internal journal
[76793.915200] EXT3-fs (dm-4): mounted filesystem with ordered data mode
[84651.608193] kjournald starting. Commit interval 5 seconds
[84651.646426] EXT3-fs (dm-4): using internal journal
[84651.651269] EXT3-fs (dm-4): mounted filesystem with ordered data mode
[84738.700610]
[84738.700625] =======================================================
[84738.708389] [ INFO: possible circular locking dependency detected ]
[84738.714663] 3.1.0-rc6 #4
[84738.717197] -------------------------------------------------------
[84738.723467] lvcreate/9492 is trying to acquire lock:
[84738.728435] (&md->suspend_lock){+.+...}, at: [<c02c73c4>]
dm_suspend+0x28/0x29c
[84738.735894]
[84738.735897] but task is already holding lock:
[84738.741740] (&journal->j_barrier){+.+...}, at: [<c01481c4>]
journal_lock_updates+0xf0/0xfc
[84738.750141]
[84738.750145] which lock already depends on the new lock.
[84738.750150]
[84738.758344]
[84738.758346] the existing dependency chain (in reverse order) is:
[84738.765839]
[84738.765842] -> #1 (&journal->j_barrier){+.+...}:
[84738.771981] [<c0052114>] check_prev_add+0x338/0x680
[84738.777492] [<c0052e4c>] validate_chain+0x4f4/0x674
[84738.782993] [<c0053714>] __lock_acquire+0x748/0x828
[84738.788496] [<c0053a28>] lock_release_non_nested+0x234/0x2bc
[84738.794780] [<c0053b98>] __lock_release+0xe8/0x1cc
[84738.800195] [<c0053cdc>] lock_release+0x60/0x70
[84738.805348] [<c04636d4>] __mutex_unlock_slowpath+0xd4/0x13c
[84738.811551] [<c046374c>] mutex_unlock+0x10/0x14
[84738.816705] [<c00c9434>] freeze_bdev+0x9c/0xa4
[84738.821783] [<c02c7438>] dm_suspend+0x9c/0x29c
[84738.826850] [<c02cc06c>] dev_suspend+0x5c/0x90
[84738.831921] [<c02ccdd4>] ctl_ioctl+0x228/0x2c8
[84738.836990] [<c02cce8c>] dm_ctl_ioctl+0x18/0x1c
[84738.842145] [<c00a9fa4>] do_vfs_ioctl+0x33c/0x38c
[84738.847483] [<c00aa034>] sys_ioctl+0x40/0x64
[84738.852378] [<c000dfc0>] ret_fast_syscall+0x0/0x3c
[84738.857801]
[84738.857804] -> #0 (&md->suspend_lock){+.+...}:
[84738.863769] [<c0051c78>] print_circular_bug+0x5c/0xfc
[84738.869445] [<c0051f10>] check_prev_add+0x134/0x680
[84738.874947] [<c0052e4c>] validate_chain+0x4f4/0x674
[84738.880450] [<c0053714>] __lock_acquire+0x748/0x828
[84738.885952] [<c0053d78>] lock_acquire+0x8c/0xa0
[84738.891107] [<c04643a8>] mutex_lock_nested+0x74/0x30c
[84738.896784] [<c02c73c4>] dm_suspend+0x28/0x29c
[84738.901851] [<c02cc06c>] dev_suspend+0x5c/0x90
[84738.906920] [<c02ccdd4>] ctl_ioctl+0x228/0x2c8
[84738.911988] [<c02cce8c>] dm_ctl_ioctl+0x18/0x1c
[84738.917143] [<c00a9fa4>] do_vfs_ioctl+0x33c/0x38c
[84738.922472] [<c00aa034>] sys_ioctl+0x40/0x64
[84738.927367] [<c000dfc0>] ret_fast_syscall+0x0/0x3c
[84738.932783]
[84738.932786] other info that might help us debug this:
[84738.932791]
[84738.940814] Possible unsafe locking scenario:
[84738.940819]
[84738.946747] CPU0 CPU1
[84738.951280] ---- ----
[84738.955810] lock(&journal->j_barrier);
[84738.959757] lock(&md->suspend_lock);
[84738.966049] lock(&journal->j_barrier);
[84738.972515] lock(&md->suspend_lock);
[84738.976287]
[84738.976290] *** DEADLOCK ***
[84738.976293]
[84738.982232] 1 lock held by lvcreate/9492:
[84738.986243] #0: (&journal->j_barrier){+.+...}, at: [<c01481c4>]
journal_lock_updates+0xf0/0xfc
[84738.995085]
[84738.995088] stack backtrace:
[84738.999455] Backtrace:
[84739.001928] [<c00114e0>] (dump_backtrace+0x0/0x110) from
[<c04624e0>] (dump_stack+0x18/0x1c)
[84739.010372] r7:dee6f908 r6:dee6f920 r5:00000000 r4:ffffffff
[84739.016090] [<c04624c8>] (dump_stack+0x0/0x1c) from [<c0051d00>]
(print_circular_bug+0xe4/0xfc)
[84739.024799] [<c0051c1c>] (print_circular_bug+0x0/0xfc) from
[<c0051f10>] (check_prev_add+0x134/0x680)
[84739.034030] [<c0051ddc>] (check_prev_add+0x0/0x680) from
[<c0052e4c>] (validate_chain+0x4f4/0x674)
[84739.043001] [<c0052958>] (validate_chain+0x0/0x674) from
[<c0053714>] (__lock_acquire+0x748/0x828)
[84739.051972] [<c0052fcc>] (__lock_acquire+0x0/0x828) from
[<c0053d78>] (lock_acquire+0x8c/0xa0)
[84739.060594] [<c0053cec>] (lock_acquire+0x0/0xa0) from [<c04643a8>]
(mutex_lock_nested+0x74/0x30c)
[84739.069478] [<c0464334>] (mutex_lock_nested+0x0/0x30c) from
[<c02c73c4>] (dm_suspend+0x28/0x29c)
[84739.078274] [<c02c739c>] (dm_suspend+0x0/0x29c) from [<c02cc06c>]
(dev_suspend+0x5c/0x90)
[84739.086463] [<c02cc010>] (dev_suspend+0x0/0x90) from [<c02ccdd4>]
(ctl_ioctl+0x228/0x2c8)
[84739.094644] r7:00004000 r6:00000006 r5:020b2560 r4:00000000
[84739.100363] [<c02ccbac>] (ctl_ioctl+0x0/0x2c8) from [<c02cce8c>]
(dm_ctl_ioctl+0x18/0x1c)
[84739.108553] [<c02cce74>] (dm_ctl_ioctl+0x0/0x1c) from [<c00a9fa4>]
(do_vfs_ioctl+0x33c/0x38c)
[84739.117090] [<c00a9c68>] (do_vfs_ioctl+0x0/0x38c) from [<c00aa034>]
(sys_ioctl+0x40/0x64)
[84739.125272] r7:00000003 r6:c138fd06 r5:020b2560 r4:df9d5600
[84739.130991] [<c00a9ff4>] (sys_ioctl+0x0/0x64) from [<c000dfc0>]
(ret_fast_syscall+0x0/0x3c)
[84739.139346] r7:00000036 r6:020b2560 r5:401c129c r4:0209e190
[84739.652392] kjournald starting. Commit interval 5 seconds
[84739.687694] EXT3-fs (dm-7): using internal journal
[84739.692513] EXT3-fs (dm-7): mounted filesystem with ordered data mode
[84808.774816] kjournald starting. Commit interval 5 seconds
[84808.810005] EXT3-fs (dm-9): using internal journal
[84808.814824] EXT3-fs (dm-9): mounted filesystem with ordered data mode
[84842.914913] kjournald starting. Commit interval 5 seconds
[84842.951118] EXT3-fs (dm-11): using internal journal
[84842.956051] EXT3-fs (dm-11): mounted filesystem with ordered data mode
[85634.390437] kjournald starting. Commit interval 5 seconds
[85634.428307] EXT3-fs (dm-13): using internal journal
[85634.433214] EXT3-fs (dm-13): mounted filesystem with ordered data mode
[87528.329271] kjournald starting. Commit interval 5 seconds
[87528.367141] EXT3-fs (dm-13): using internal journal
[87528.372050] EXT3-fs (dm-13): mounted filesystem with ordered data mode
[88075.274597] kjournald starting. Commit interval 5 seconds
[88075.312460] EXT3-fs (dm-13): using internal journal
[88075.317400] EXT3-fs (dm-13): mounted filesystem with ordered data mode
[88524.830124] kjournald starting. Commit interval 5 seconds
[88524.867964] EXT3-fs (dm-13): using internal journal
[88524.872872] EXT3-fs (dm-13): mounted filesystem with ordered data mode
[88775.331566] Unable to handle kernel NULL pointer dereference at
virtual address 00000000
[88775.339749] pgd = d5170000
[88775.342460] [00000000] *pgd=80549831, *pte=00000000, *ppte=00000000
[88775.348788] Internal error: Oops: 817 [#1]
[88775.352888] Modules linked in: mma8450 input_polldev
[88775.357896] CPU: 0 Not tainted (3.1.0-rc6 #4)
[88775.362621] PC is at mempool_destroy+0x20/0x30
[88775.367075] LR is at dm_kcopyd_client_destroy+0x120/0x134
[88775.372479] pc : [<c006d1a8>] lr : [<c02cd850>] psr: 80000013
[88775.372486] sp : cbc03e18 ip : cbc03e28 fp : cbc03e24
[88775.383973] r10: c02cc4e8 r9 : cbc02000 r8 : df9da3f8
[88775.389202] r7 : 00000000 r6 : c0c59468 r5 : 00000000 r4 : df9da3c0
[88775.395734] r3 : 00000000 r2 : 00000008 r1 : cbc02000 r0 : dfac2280
[88775.402268] Flags: Nzcv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user
[88775.409409] Control: 10c5387d Table: 85170019 DAC: 00000015
[88775.415159] Process lvremove (pid: 26882, stack limit = 0xcbc022e8)
[88775.421431] Stack: (0xcbc03e18 to 0xcbc04000)
[88775.425792] 3e00:
cbc03e64 cbc03e28
[88775.433979] 3e20: c02cd850 c006d194 cbc02000 c02cc4e8 cbc03e54
cbc03e40 c006d184 c0096ea4
[88775.442166] 3e40: dfaa4e00 e2ea2040 c0c59468 00000000 0000020c
c02cc4e8 cbc03e8c cbc03e68
[88775.450353] 3e60: c02d45a8 c02cd73c 00000000 dfaa4e00 db6b5500
00000038 00000001 e2ea2040
[88775.458540] 3e80: cbc03eac cbc03e90 c02c9560 c02d44b0 deda4800
db6b5500 deda4884 def66dc0
[88775.466727] 3ea0: cbc03ecc cbc03eb0 c02c7764 c02c94f4 deda4800
e360d000 00000000 def66dc0
[88775.474913] 3ec0: cbc03edc cbc03ed0 c02c77ac c02c7644 cbc03efc
cbc03ee0 c02cc5bc c02c77a4
[88775.483100] 3ee0: 00000000 01a92348 00000004 00004000 cbc03f44
cbc03f00 c02ccdd4 c02cc4f4
[88775.491287] 3f00: c00b6374 c00b6278 cbc03f54 cbc03f18 00000004
00000015 00000000 e360d000
[88775.499474] 3f20: dec6de40 01a92348 c138fd04 00000005 c000e184
00000000 cbc03f54 cbc03f48
[88775.507661] 3f40: c02cce8c c02ccbb8 cbc03f7c cbc03f58 c00a9fa4
c02cce80 cbc03f84 cbc03f68
[88775.515848] 3f60: dec6de40 01a92348 c138fd04 00000005 cbc03fa4
cbc03f80 c00aa034 c00a9c74
[88775.524035] 3f80: c00508c8 00000000 01a8b1c8 4014529c 01a92348
00000036 00000000 cbc03fa8
[88775.532221] 3fa0: c000dfc0 c00aa000 01a8b1c8 4014529c 00000005
c138fd04 01a92348 000012ec
[88775.540408] 3fc0: 01a8b1c8 4014529c 01a92348 00000036 01a92378
0000131c 0005cc80 40144834
[88775.548594] 3fe0: 40143364 bee1d048 40140014 4031a70c 20000010
00000005 8fffe821 8fffec21
[88775.556775] Backtrace:
[88775.559246] [<c006d188>] (mempool_destroy+0x0/0x30) from
[<c02cd850>] (dm_kcopyd_client_destroy+0x120/0x134)
[88775.569092] [<c02cd730>] (dm_kcopyd_client_destroy+0x0/0x134) from
[<c02d45a8>] (snapshot_dtr+0x104/0x164)
[88775.578764] [<c02d44a4>] (snapshot_dtr+0x0/0x164) from [<c02c9560>]
(dm_table_destroy+0x78/0xec)
[88775.587555] r7:e2ea2040 r6:00000001 r5:00000038 r4:db6b5500
[88775.593274] [<c02c94e8>] (dm_table_destroy+0x0/0xec) from
[<c02c7764>] (__dm_destroy+0x12c/0x148)
[88775.602150] r7:def66dc0 r6:deda4884 r5:db6b5500 r4:deda4800
[88775.607868] [<c02c7638>] (__dm_destroy+0x0/0x148) from [<c02c77ac>]
(dm_destroy+0x14/0x18)
[88775.616137] r7:def66dc0 r6:00000000 r5:e360d000 r4:deda4800
[88775.621858] [<c02c7798>] (dm_destroy+0x0/0x18) from [<c02cc5bc>]
(dev_remove+0xd4/0xec)
[88775.629874] [<c02cc4e8>] (dev_remove+0x0/0xec) from [<c02ccdd4>]
(ctl_ioctl+0x228/0x2c8)
[88775.637968] r7:00004000 r6:00000004 r5:01a92348 r4:00000000
[88775.643686] [<c02ccbac>] (ctl_ioctl+0x0/0x2c8) from [<c02cce8c>]
(dm_ctl_ioctl+0x18/0x1c)
[88775.651886] [<c02cce74>] (dm_ctl_ioctl+0x0/0x1c) from [<c00a9fa4>]
(do_vfs_ioctl+0x33c/0x38c)
[88775.660423] [<c00a9c68>] (do_vfs_ioctl+0x0/0x38c) from [<c00aa034>]
(sys_ioctl+0x40/0x64)
[88775.668605] r7:00000005 r6:c138fd04 r5:01a92348 r4:dec6de40
[88775.674330] [<c00a9ff4>] (sys_ioctl+0x0/0x64) from [<c000dfc0>]
(ret_fast_syscall+0x0/0x3c)
[88775.682685] r7:00000036 r6:01a92348 r5:4014529c r4:01a8b1c8
[88775.688403] Code: e5903024 e1530002 0a000002 e3a03000 (e5833000)
[88775.694552] ---[ end trace 6c1efa42e6c1017d ]---
You might be able to track downstream bug at:
<http://bugs.debian.org/642235>
Please CC me or 642235 at bugs.debian.org on reply as I am not subscribed
to this mailing list.
Best regards,
--
Héctor Orón -.. . -... .. .- -. -.. . ...- . .-.. --- .--. . .-.
More information about the dm-devel
mailing list