[dm-devel] lvmetad doesn't terminate with SIGTERM if thin volume used

[james harvey]

Does it matter at all if lvmetad shuts down gracefully?

Can I safely just have systemd right off the bat send a SIGKILL?

Most things I wouldn't ask about, but I'm wondering if this is PURELY
a caching daemon where gracefully shutting down doesn't really do
anything.

On Wed, Aug 3, 2016 at 10:51 PM, james harvey <jamespharvey20 at gmail.com> wrote:
> After upgrading to systemd v231, my shutdowns/reboots have a 90 second
> delay at the very end.  Linux kvm 4.6.4-1.
>
> After I looked into it, I found it's due to lvmetad never terminating
> when receiving a SIGTERM, and after 90 seconds, systemd performs a
> SIGKILL.
>
> systemd 231 (commit d4506129) changed the timeout on sending a SIGKILL
> after a SIGTERM from 10 seconds to 90 seconds.  I think this bug has
> been around for quite a while, because I've noticed shutdowns had
> about a 10 second delay at the same spot that now has a 90 second
> delay.
>
> With lvmetad running with "-l all", a systemd debug dmesg log through
> shutdown is attached, after running it through "grep -i lvm2".  The
> full (4MB) version is here:
> http://45.63.106.241/share/lvm2-lvmetad.shutdown-log2.txt
>
> This also happens if I attempt stopping lvm2-lvmetad.  Attached is
> information showing that.
>
> Also attached is the minimal steps I used to cause the problem, using
> one disk and EXT4.
>
> If during the install I combine the 2 lvcreate commands into a single
> one without using thin pools, then lvmetad terminates pretty much
> immediately with SIGTERM.
>
> ==============================
>
> # systemctl status lvm2-lvmetad
> ● lvm2-lvmetad.service - LVM2 metadata daemon
>    Loaded: loaded (/usr/lib/systemd/system/lvm2-lvmetad.service;
> disabled; vendor preset: disabled)
>    Active: active (running) since Wed 2016-08-03 21:36:54 EDT; 1min 14s ago
>      Docs: man:lvmetad(8)
>  Main PID: 398 (lvmetad)
>     Tasks: 2 (limit: 4915)
>    CGroup: /system.slice/lvm2-lvmetad.service
>            └─398 /usr/bin/lvmetad -f
>
> # systemctl stop lvm2-lvmetad
> {{{ after 90 seconds }}}
> Warning: Stopping lvm2-lvmetad.service, but it can still be activated by:
>   lvm2-lvmetad.socket
> # systemctl status lvm2-lvmetad
> ● lvm2-lvmetad.service - LVM2 metadata daemon
>    Loaded: loaded (/usr/lib/systemd/system/lvm2-lvmetad.service;
> disabled; vendor preset: disabled)
>    Active: failed (Result: signal) since Wed 2016-08-03 21:40:33 EDT; 12s ago
>      Docs: man:lvmetad(8)
>   Process: 398 ExecStart=/usr/bin/lvmetad -f (code=killed, signal=KILL)
>  Main PID: 398 (code=killed, signal=KILL)
>
> Aug 03 21:36:54 terra systemd[1]: Started LVM2 metadata daemon.
> Aug 03 21:39:03 terra systemd[1]: Stopping LVM2 metadata daemon...
> Aug 03 21:39:03 terra lvmetad[398]: Failed to accept connection.
> Aug 03 21:40:33 terra systemd[1]: lvm2-lvmetad.service: State
> 'stop-sigterm' timed out. Killing.
> Aug 03 21:40:33 terra systemd[1]: lvm2-lvmetad.service: Killing
> process 398 (lvmetad) with signal SIGKILL.
> Aug 03 21:40:33 terra systemd[1]: lvm2-lvmetad.service: Main process
> exited, code=killed, status=9/KILL
> Aug 03 21:40:33 terra systemd[1]: Stopped LVM2 metadata daemon.
> Aug 03 21:40:33 terra systemd[1]: lvm2-lvmetad.service: Unit entered
> failed state.
> Aug 03 21:40:33 terra systemd[1]: lvm2-lvmetad.service: Failed with
> result 'signal'.
>
> ====================
>
> /dev/sda1 3.5G Linux filesystem
> /dev/sda2 4.5TB Linux LVM
>
> { Setup LVM and filesystems }
> # mkfs.ext4 -L boot /dev/sda1
> # pvcreate /dev/sda2
> # vgcreate disk1 /dev/sda2
> { Merging these 2 lvcreates, removing the thin volume usage makes
> lvm2-lvmetad properly terminate on SIGTERM }
> # lvcreate --size 500G --thinpool disk1thin disk1
> # lvcreate --virtualsize 100G --name root disk1/disk1thin
> # mkfs.ext4 -L /mnt /dev/disk1/main
> # mount /dev/disk1/main /mnt
> # mkdir /mnt/boot
> # mount /dev/sda1 /mnt/boot
>
> { Install Arch Linux }
> # vi /etc/pacman.d/mirrorlist
> # pacstrap -i /mnt base syslinux gptfdisk lvm2
> # arch-chroot /mnt
> # vi /etc/locale.gen
> # locale-gen
> # locale > /etc/locale.conf
> # vi /etc/nsswitch.conf
> # systemctl enable systemd-resolved systemd-networkd
> # ln -s /usr/share/zoneinfo/America/Detroit /etc/localtime
> # hwclock --utc --systohc
> # passwd
> { Add lvm2 between block and filesystems }
> # vi /etc/mkinitcpio.conf
> # mkinitcpio -p linux
> # echo hostname > /etc/hostname
> # vi /etc/systemd/network/enp31s0.network
> # syslinux-install_update -i -a -m
> # vi /boot/syslinux/syslinux.cfg
>
> { After Reboot }
> # vi /etc/fstab