[linux-lvm] Device mapper does not really remove the references to removed LV.

[Germain Maurice]

Hi everybody,

I'm facing to some problems in my virtualization cluster and i need some help to prevent it occurring again.

I know that LVM is not cluster aware and we have to use it with lot of caution.
However, i have a shared LVM storage between 6 nodes. I use KVM hosts.

Two days ago i rebooted my whole cluster in order to get the same LVM metadata and the same dmsetup table over the cluster. That's ok.

"vgck SATA6To" is ok.

Right now, I have a problem when deleting an LV.

On the node where the VM lives, i did the lvremove command (lv name : SATA6To/vm-306-disk-3), i have a good state of LVM :

node5:~# lvdisplay -C | grep -- "-306"
  vm-306-disk-1 SATA6To     -wi-ao  10.00G                                      
  vm-306-disk-2 SATA6To     -wi-ao   5.00G     
node5:~# dmsetup table | grep -- "-306"
SATA6To-vm--306--disk--1: 0 20971520 linear 152:16 2231370112
SATA6To-vm--306--disk--2: 0 10485760 linear 152:16 2252341632
node5:~# ls /dev/SATA6To/vm-306-disk-*
/dev/SATA6To/vm-306-disk-1  /dev/SATA6To/vm-306-disk-2
node5:~# ls /dev/mapper/SATA6To-vm--306--disk--*
/dev/mapper/SATA6To-vm--306--disk--1  /dev/mapper/SATA6To-vm--306--disk--2


If i did a "vgscan or lvscan" on the rest of the cluster, LVM well sees that the LV has been removed, however device-mapper still have devices for it :

node3:~# lvdisplay -C | grep -- "-306"
  vm-306-disk-1 SATA6To     -wi-a-  10.00G                                      
  vm-306-disk-2 SATA6To     -wi-a-   5.00G
node3:~# dmsetup table  | grep -- "-306"
SATA6To-vm--306--disk--1: 0 20971520 linear 152:32 2231370112
SATA6To-vm--306--disk--3: 0 8388608 linear 152:32 2262827392
SATA6To-vm--306--disk--2: 0 10485760 linear 152:32 2252341632
node3:~# ls /dev/SATA6To/vm-306-disk-*
/dev/SATA6To/vm-306-disk-1  /dev/SATA6To/vm-306-disk-2	/dev/SATA6To/vm-306-disk-3
node3:~# ls /dev/mapper/SATA6To-vm--306--disk--*
/dev/mapper/SATA6To-vm--306--disk--1  /dev/mapper/SATA6To-vm--306--disk--3
/dev/mapper/SATA6To-vm--306--disk--2


Any idea how to clear all references to the LV previously removed ?

And by the way, the good way to do the best practices when i change something in LVM and to take it in account in the other nodes ?

Is it advised to disable lvm cache (/etc/lvm/cache/.cache) ?

Thank you for your precious help !
Germain