[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

[linux-lvm] Cannot online resize in a cluster: Invalid argument



Hello all,

I have a three node cluster that's managed using corosync and openais.
We are using clvmd to propagate lvm meta-data across nodes. clvmd is
started by corosync with the "-Iopenais -d3 -t15" arguments.

When I attempt to modify a logical volume, either through resizing or
renaming, the request fails with:
cluster request failed: Invalid argument
Failed to suspend clone_os

The detail output is:
qa-xen-0:/ # lvresize /dev/qa-xen-vm-store/clone_os -L 10G -vvvv
#lvmcmdline.c:1045         Processing: lvresize
/dev/qa-xen-vm-store/clone_os -L 10G -vvvv
#config/config.c:996       Setting global/locking_type to 3
#config/config.c:1001       global/wait_for_locks not found in config:
defaulting to 1
#locking/locking.c:271       Cluster locking selected.
#lvresize.c:773     Finding volume group qa-xen-vm-store
#locking/cluster_locking.c:497       Locking VG V_qa-xen-vm-store PW (VG) (0x4)
#device/dev-cache.c:331         /dev/block/1:0: Added to device cache
...
#filters/filter-composite.c:31         Using /dev/mapper/qa-xen-vm-store
#device/dev-io.c:535         Opened /dev/mapper/qa-xen-vm-store RO
#device/dev-io.c:137         /dev/mapper/qa-xen-vm-store: block size
is 4096 bytes
#label/label.c:162       /dev/mapper/qa-xen-vm-store: lvm2 label
detected at sector 1
#cache/lvmcache.c:1226         lvmcache: /dev/mapper/qa-xen-vm-store:
now in VG #orphans_lvm2 (#orphans_lvm2)
#format_text/format-text.c:1187         /dev/mapper/qa-xen-vm-store:
Found metadata at 57856 size 27829 (in area at 4096 size 192512) for
qa-xen-vm-store (HU1eAV-0WUq-Xxtb-emD7-9mz1-ZDwj-ba4B4X)
#cache/lvmcache.c:1226         lvmcache: /dev/mapper/qa-xen-vm-store:
now in VG qa-xen-vm-store with 1 mdas
#cache/lvmcache.c:1003         lvmcache: /dev/mapper/qa-xen-vm-store:
setting qa-xen-vm-store VGID to HU1eAV0WUqXxtbemD79mz1ZDwjba4B4X
#cache/lvmcache.c:1263         lvmcache: /dev/mapper/qa-xen-vm-store:
VG qa-xen-vm-store: Set creation host to qa-xen-0.
#format_text/archiver.c:128     Archiving volume group
"qa-xen-vm-store" metadata (seqno 149).
  Extending logical volume clone_os to 10.00 GiB
#metadata/lv_manip.c:2492       Extending segment type, striped
#metadata/pv_map.c:55         Allowing allocation on
/dev/mapper/qa-xen-vm-store start PE 180448 length 17920
#metadata/pv_map.c:55         Allowing allocation on
/dev/mapper/qa-xen-vm-store start PE 221664 length 19138
#metadata/lv_manip.c:1948         Trying allocation using contiguous policy.
#metadata/lv_manip.c:1562         Still need 1536 total extents:
#metadata/lv_manip.c:1565           1 (1 data/0 parity) parallel areas
of 1536 extents each
#metadata/lv_manip.c:1567           0 mirror logs of 0 extents each
#metadata/lv_manip.c:1948         Trying allocation using cling policy.
#metadata/lv_manip.c:1562         Still need 1536 total extents:
#metadata/lv_manip.c:1565           1 (1 data/0 parity) parallel areas
of 1536 extents each
#metadata/lv_manip.c:1567           0 mirror logs of 0 extents each
#metadata/lv_manip.c:1261         Considering allocation area 0 as
/dev/mapper/qa-xen-vm-store start PE 221664 length 19138 leaving 0.
#metadata/lv_manip.c:1044         Allocating parallel area 0 on
/dev/mapper/qa-xen-vm-store start PE 221664 length 1536.
#locking/cluster_locking.c:497       Locking VG P_qa-xen-vm-store PW
(VG|CACHE) (0x104)
#device/dev-io.c:588         Closed /dev/mapper/qa-xen-vm-store
#device/dev-io.c:535         Opened /dev/mapper/qa-xen-vm-store RW
#device/dev-io.c:137         /dev/mapper/qa-xen-vm-store: block size
is 4096 bytes
#format_text/format-text.c:629         Writing qa-xen-vm-store
metadata to /dev/mapper/qa-xen-vm-store at 86016 len 27955
#format_text/format-text.c:734         Pre-Committing qa-xen-vm-store
metadata (150) to /dev/mapper/qa-xen-vm-store header at 4096
#mm/memlock.c:402         Entering critical section (locking for suspend).
#mm/memlock.c:378         Lock:   Memlock counters: locked:0
critical:1 daemon:0 suspended:0
#mm/memlock.c:330       Locking memory
#mm/memlock.c:212         mlock        772KiB       400000 -
4c1000 r-xp 00000000 08:01 142893                             /bin/lvm
#mm/memlock.c:212         mlock         32KiB       6c1000 -
6c9000 rw-p 000c1000 08:01 142893                             /bin/lvm
#mm/memlock.c:212         mlock      10808KiB       6c9000 -
1157000 rw-p 00000000 00:00 0                                  [heap]
#mm/memlock.c:186         mlock default filter '/libncurses.so.'
matches '7f076a450000-7f076a49e000 r-xp 00000000 08:01 170277
           /lib64/libncurses.so.5.0.4': Skipping.
#mm/memlock.c:168         mlock area unreadable
7f076a49e000-7f076a69e000 ---p 0004e000 08:01 170277
  /lib64/libncurses.so.5.0.4 : Skipping.
#mm/memlock.c:186         mlock default filter '/libncurses.so.'
matches '7f076a69e000-7f076a6ac000 rw-p 0004e000 08:01 170277
           /lib64/libncurses.so.5.0.4': Skipping.
#mm/memlock.c:212         mlock          4KiB 7f076a6ac000 -
7f076a6ad000 rw-p 00000000 00:00 0
#mm/memlock.c:212         mlock       1320KiB 7f076a6ad000 -
7f076a7f7000 r-xp 00000000 08:01 170005
/lib64/libc-2.9.so
#mm/memlock.c:168         mlock area unreadable
7f076a7f7000-7f076a9f6000 ---p 0014a000 08:01 170005
  /lib64/libc-2.9.so : Skipping.
#mm/memlock.c:212         mlock         16KiB 7f076a9f6000 -
7f076a9fa000 r--p 00149000 08:01 170005
/lib64/libc-2.9.so
#mm/memlock.c:212         mlock          4KiB 7f076a9fa000 -
7f076a9fb000 rw-p 0014d000 08:01 170005
/lib64/libc-2.9.so
#mm/memlock.c:212         mlock         20KiB 7f076a9fb000 -
7f076aa00000 rw-p 00000000 00:00 0
#mm/memlock.c:186         mlock default filter '/libreadline.so.'
matches '7f076aa00000-7f076aa33000 r-xp 00000000 08:01 170244
           /lib64/libreadline.so.5.2': Skipping.
#mm/memlock.c:168         mlock area unreadable
7f076aa33000-7f076ac33000 ---p 00033000 08:01 170244
  /lib64/libreadline.so.5.2 : Skipping.
#mm/memlock.c:186         mlock default filter '/libreadline.so.'
matches '7f076ac33000-7f076ac3b000 rw-p 00033000 08:01 170244
           /lib64/libreadline.so.5.2': Skipping.
#mm/memlock.c:212         mlock          4KiB 7f076ac3b000 -
7f076ac3c000 rw-p 00000000 00:00 0
#mm/memlock.c:212         mlock        144KiB 7f076ac3c000 -
7f076ac60000 r-xp 00000000 08:01 170002
/lib64/libdevmapper.so.1.02
#mm/memlock.c:168         mlock area unreadable
7f076ac60000-7f076ae5f000 ---p 00024000 08:01 170002
  /lib64/libdevmapper.so.1.02 : Skipping.
#mm/memlock.c:212         mlock          8KiB 7f076ae5f000 -
7f076ae61000 rw-p 00023000 08:01 170002
/lib64/libdevmapper.so.1.02
#mm/memlock.c:186         mlock default filter '/libdl-' matches
'7f076ae61000-7f076ae63000 r-xp 00000000 08:01 170016
   /lib64/libdl-2.9.so': Skipping.
#mm/memlock.c:168         mlock area unreadable
7f076ae63000-7f076b063000 ---p 00002000 08:01 170016
  /lib64/libdl-2.9.so : Skipping.
#mm/memlock.c:186         mlock default filter '/libdl-' matches
'7f076b063000-7f076b064000 r--p 00002000 08:01 170016
   /lib64/libdl-2.9.so': Skipping.
#mm/memlock.c:186         mlock default filter '/libdl-' matches
'7f076b064000-7f076b065000 rw-p 00003000 08:01 170016
   /lib64/libdl-2.9.so': Skipping.
#mm/memlock.c:212         mlock        116KiB 7f076b065000 -
7f076b082000 r-xp 00000000 08:01 170025
/lib64/ld-2.9.so
#mm/memlock.c:212         mlock         16KiB 7f076b279000 -
7f076b27d000 rw-p 00000000 00:00 0
#mm/memlock.c:212         mlock          4KiB 7f076b280000 -
7f076b281000 rw-p 00000000 00:00 0
#mm/memlock.c:212         mlock          4KiB 7f076b281000 -
7f076b282000 r--p 0001c000 08:01 170025
/lib64/ld-2.9.so
#mm/memlock.c:212         mlock          4KiB 7f076b282000 -
7f076b283000 rw-p 0001d000 08:01 170025
/lib64/ld-2.9.so
#mm/memlock.c:212         mlock        268KiB 7fffa086b000 -
7fffa08ae000 rw-p 00000000 00:00 0                          [stack]
#mm/memlock.c:176         mlock ignore filter '[vdso]' matches
'7fffa0940000-7fffa0941000 r-xp 00000000 00:00 0
   [vdso]': Skipping.
#mm/memlock.c:176         mlock ignore filter '[vsyscall]' matches
'ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0
   [vsyscall]': Skipping.
#mm/memlock.c:297         Locked 13869056 bytes
#locking/cluster_locking.c:497       Locking LV
HU1eAV0WUqXxtbemD79mz1ZDwjba4B4Xjr2bZasokY126nki9bO38iFTHJDbsAyj PW
(LV|NONBLOCK|CLUSTER) (0x9c)
#locking/cluster_locking.c:160   cluster request failed: Invalid argument
#locking/locking.c:396         <backtrace>
#locking/locking.c:465         <backtrace>
#lvresize.c:729   Failed to suspend clone_os
#format_text/format-text.c:738         Wiping pre-committed
qa-xen-vm-store metadata from /dev/mapper/qa-xen-vm-store header at
4096
#locking/cluster_locking.c:497       Locking VG P_qa-xen-vm-store PW
(VG|CACHE) (0x104)
#locking/cluster_locking.c:497       Locking VG P_qa-xen-vm-store CR
(VG|HOLD|CACHE) (0x121)
#locking/cluster_locking.c:425       Requesting backup of VG metadata
for qa-xen-vm-store
#format_text/archiver.c:395     Creating volume group backup
"/etc/lvm/backup/qa-xen-vm-store" (seqno 150).
#format_text/format-text.c:929         Writing qa-xen-vm-store
metadata to /etc/lvm/backup/.lvm_qa-xen-0_30840_1908312135
#format_text/format-text.c:973         Committing qa-xen-vm-store metadata (150)
#format_text/format-text.c:974         Renaming
/etc/lvm/backup/qa-xen-vm-store.tmp to /etc/lvm/backup/qa-xen-vm-store
#mm/memlock.c:389         Unlock: Memlock counters: locked:1
critical:1 daemon:0 suspended:0
#locking/cluster_locking.c:419       Requesting sync names.
#locking/cluster_locking.c:497       Locking VG V_qa-xen-vm-store UN (VG) (0x6)
#device/dev-io.c:588         Closed /dev/mapper/qa-xen-vm-store
#metadata/vg.c:68         Freeing VG qa-xen-vm-store at 0x71f350.

The only patch we are applying to lvm2 is removal of the glum code
that caused lvm to hang when a node was offline. In all other respects
this is the stock version of lvm from
ftp://sources.redhat.com/pub/lvm2/ .

I thought this might be related to
https://bugzilla.redhat.com/show_bug.cgi?id=364081 but I am not that
experienced with C coding and cannot seem to track down where the
problem lies.

Any help on this would be greatly appreciated, as the ability to
online resize/rename is a critical feature that is currently
non-functional for me.

Thanks,
Hal Martin


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]