[Linux-cluster] locking_type

Thu Oct 11 11:19:09 UTC 2012

On Thursday, October 11, 2012 11:36:05 AM Alasdair G Kergon wrote:
> On Thu, Oct 11, 2012 at 12:03:02PM +0200, Ali Bendriss wrote:
> > Since the updrade to current setup, I'm only able to run clvmd if it is
> > compiled using "--with-cluster=shared" and setting the locking_type = 2 in
> > clvm.conf
> 
> What is the error you get?
> 
Starting the node with locking type = 3 and clvmd compiled using with-
cluster=internal , I've got no error:
clvmd: Cluster LVM daemon started - connected to CMAN

but running for example vgscan desn't work:
/sbin/vgscan
  connect() failed on local socket: No such file or directory
  Internal cluster locking initialisation failed.
  WARNING: Falling back to local file-based locking.
  Volume Groups with the clustered attribute will be inaccessible.
  Reading all physical volumes.  This may take a while...
  Skipping clustered volume group samba4
  Skipping clustered volume group ctdb
  Skipping clustered volume group shared
  Found volume group "main" using metadata type lvm2

the same command is working using clvmd compiled using the shared locking.

more log below

> I don't think we stopped this working intentionally, but admittedly it's
> not a configuration we test very often.
> 
I was thinking that "--with-cluster=internal", was the recommended 
configuration.

> Alasdair

--------------------------------------------------------------------------------------------
(using locking type = 3) and calling vgscan

#clvmd -d 1 
CLVMD[9d22b740]: Oct 11 13:03:51 CLVMD started
CLVMD[9d22b740]: Oct 11 13:03:51 Connected to CMAN
CLVMD[9d22b740]: Oct 11 13:03:51 CMAN initialisation complete
CLVMD[9d22b740]: Oct 11 13:03:51 Created DLM lockspace for CLVMD.
CLVMD[9d22b740]: Oct 11 13:03:51 DLM initialisation complete
CLVMD[9d22b740]: Oct 11 13:03:51 Cluster ready, doing some more initialisation
CLVMD[9d22b740]: Oct 11 13:03:51 starting LVM thread
CLVMD[9d22a700]: Oct 11 13:03:51 LVM thread function started
WARNING: Locking disabled. Be careful! This could corrupt your metadata.
Incorrect metadata area header checksum on /dev/sdd at offset 4096
CLVMD[9d22a700]: Oct 11 13:03:51 getting initial lock for 
ve26qDQ7hcpgDH2fw19GFZkbKgTadysCNUNjh9w8HFdbVvQLBjZidl8QseraUBc0
CLVMD[9d22a700]: Oct 11 13:03:51 sync_lock: 
've26qDQ7hcpgDH2fw19GFZkbKgTadysCNUNjh9w8HFdbVvQLBjZidl8QseraUBc0' mode:1 
flags=1
CLVMD[9d22a700]: Oct 11 13:03:51 sync_lock: returning lkid 1
CLVMD[9d22a700]: Oct 11 13:03:51 getting initial lock for 
ve26qDQ7hcpgDH2fw19GFZkbKgTadysCAXDuTvYJ4ambKnLALpOffDSxPrjHliO0
CLVMD[9d22a700]: Oct 11 13:03:51 sync_lock: 
've26qDQ7hcpgDH2fw19GFZkbKgTadysCAXDuTvYJ4ambKnLALpOffDSxPrjHliO0' mode:1 
flags=1
CLVMD[9d22a700]: Oct 11 13:03:51 sync_lock: returning lkid 2
CLVMD[9d22a700]: Oct 11 13:03:51 getting initial lock for 
ByYnMJeHNgSIBuJENA2WLMe148edovbofR4f9clPHk2BUveeSUstcSEJzcOHt2BE
CLVMD[9d22a700]: Oct 11 13:03:51 sync_lock: 
'ByYnMJeHNgSIBuJENA2WLMe148edovbofR4f9clPHk2BUveeSUstcSEJzcOHt2BE' mode:1 
flags=1
CLVMD[9d22a700]: Oct 11 13:03:51 sync_lock: returning lkid 3
CLVMD[9d22a700]: Oct 11 13:03:51 getting initial lock for 
GkJTcHFLl6YNc8M0QN7yYA0nWwhzHKjx6QSKUI7riC86QhzIX98cmu8rL4lHXJlO
CLVMD[9d22a700]: Oct 11 13:03:51 sync_lock: 
'GkJTcHFLl6YNc8M0QN7yYA0nWwhzHKjx6QSKUI7riC86QhzIX98cmu8rL4lHXJlO' mode:1 
flags=1
CLVMD[9d22a700]: Oct 11 13:03:51 sync_lock: returning lkid 4
CLVMD[9d22a700]: Oct 11 13:03:51 getting initial lock for 
GkJTcHFLl6YNc8M0QN7yYA0nWwhzHKjxS36cpLJRPxWYtjPQChMIQZW7Zxx97aGL
CLVMD[9d22a700]: Oct 11 13:03:51 sync_lock: 
'GkJTcHFLl6YNc8M0QN7yYA0nWwhzHKjxS36cpLJRPxWYtjPQChMIQZW7Zxx97aGL' mode:1 
flags=1
CLVMD[9d22a700]: Oct 11 13:03:51 sync_lock: returning lkid 5
CLVMD[9d22a700]: Oct 11 13:03:51 getting initial lock for 
GkJTcHFLl6YNc8M0QN7yYA0nWwhzHKjx7Vov15UStgsS1tgOISASG7bPjYf7NpYO
CLVMD[9d22a700]: Oct 11 13:03:51 sync_lock: 
'GkJTcHFLl6YNc8M0QN7yYA0nWwhzHKjx7Vov15UStgsS1tgOISASG7bPjYf7NpYO' mode:1 
flags=1
CLVMD[9d22a700]: Oct 11 13:03:51 sync_lock: returning lkid 6
CLVMD[9d22a700]: Oct 11 13:03:51 getting initial lock for 
GkJTcHFLl6YNc8M0QN7yYA0nWwhzHKjx1OWkvGiyJGfz9u5Pcedbzj4hnT2Q6TY0
CLVMD[9d22a700]: Oct 11 13:03:51 sync_lock: 
'GkJTcHFLl6YNc8M0QN7yYA0nWwhzHKjx1OWkvGiyJGfz9u5Pcedbzj4hnT2Q6TY0' mode:1 
flags=1
CLVMD[9d22a700]: Oct 11 13:03:51 sync_lock: returning lkid 7
CLVMD[9d22a700]: Oct 11 13:03:51 Sub thread ready for work.
CLVMD[9d22b740]: Oct 11 13:03:51 clvmd ready for work
CLVMD[9d22b740]: Oct 11 13:03:51 Using timeout of 60 seconds
CLVMD[9d22a700]: Oct 11 13:03:51 LVM thread waiting for work

-------------------------------------------------------------------------------------------------------------------------------------
using locking type = 2 and calling vgscan
# clvmd -d 1 

CLVMD[6d153740]: Oct 11 13:07:57 CLVMD started
CLVMD[6d153740]: Oct 11 13:07:57 Connected to CMAN
CLVMD[6d153740]: Oct 11 13:07:57 CMAN initialisation complete
CLVMD[6d153740]: Oct 11 13:07:57 Created DLM lockspace for CLVMD.
CLVMD[6d153740]: Oct 11 13:07:57 DLM initialisation complete
CLVMD[6d153740]: Oct 11 13:07:57 Cluster ready, doing some more initialisation
CLVMD[6d153740]: Oct 11 13:07:57 starting LVM thread
CLVMD[6d152700]: Oct 11 13:07:57 LVM thread function started
WARNING: Locking disabled. Be careful! This could corrupt your metadata.
Incorrect metadata area header checksum on /dev/sdd at offset 4096
CLVMD[6d152700]: Oct 11 13:07:57 getting initial lock for 
ve26qDQ7hcpgDH2fw19GFZkbKgTadysCNUNjh9w8HFdbVvQLBjZidl8QseraUBc0
CLVMD[6d152700]: Oct 11 13:07:57 sync_lock: 
've26qDQ7hcpgDH2fw19GFZkbKgTadysCNUNjh9w8HFdbVvQLBjZidl8QseraUBc0' mode:1 
flags=1
CLVMD[6d152700]: Oct 11 13:07:57 sync_lock: returning lkid 1
CLVMD[6d152700]: Oct 11 13:07:57 getting initial lock for 
ve26qDQ7hcpgDH2fw19GFZkbKgTadysCAXDuTvYJ4ambKnLALpOffDSxPrjHliO0
CLVMD[6d152700]: Oct 11 13:07:57 sync_lock: 
've26qDQ7hcpgDH2fw19GFZkbKgTadysCAXDuTvYJ4ambKnLALpOffDSxPrjHliO0' mode:1 
flags=1
CLVMD[6d152700]: Oct 11 13:07:57 sync_lock: returning lkid 2
CLVMD[6d152700]: Oct 11 13:07:57 getting initial lock for 
ByYnMJeHNgSIBuJENA2WLMe148edovbofR4f9clPHk2BUveeSUstcSEJzcOHt2BE
CLVMD[6d152700]: Oct 11 13:07:57 sync_lock: 
'ByYnMJeHNgSIBuJENA2WLMe148edovbofR4f9clPHk2BUveeSUstcSEJzcOHt2BE' mode:1 
flags=1
CLVMD[6d152700]: Oct 11 13:07:57 sync_lock: returning lkid 3
CLVMD[6d152700]: Oct 11 13:07:57 getting initial lock for 
GkJTcHFLl6YNc8M0QN7yYA0nWwhzHKjx6QSKUI7riC86QhzIX98cmu8rL4lHXJlO
CLVMD[6d152700]: Oct 11 13:07:57 sync_lock: 
'GkJTcHFLl6YNc8M0QN7yYA0nWwhzHKjx6QSKUI7riC86QhzIX98cmu8rL4lHXJlO' mode:1 
flags=1
CLVMD[6d152700]: Oct 11 13:07:57 sync_lock: returning lkid 4
CLVMD[6d152700]: Oct 11 13:07:57 getting initial lock for 
GkJTcHFLl6YNc8M0QN7yYA0nWwhzHKjxS36cpLJRPxWYtjPQChMIQZW7Zxx97aGL
CLVMD[6d152700]: Oct 11 13:07:57 sync_lock: 
'GkJTcHFLl6YNc8M0QN7yYA0nWwhzHKjxS36cpLJRPxWYtjPQChMIQZW7Zxx97aGL' mode:1 
flags=1
CLVMD[6d152700]: Oct 11 13:07:57 sync_lock: returning lkid 5
CLVMD[6d152700]: Oct 11 13:07:57 getting initial lock for 
GkJTcHFLl6YNc8M0QN7yYA0nWwhzHKjx7Vov15UStgsS1tgOISASG7bPjYf7NpYO
CLVMD[6d152700]: Oct 11 13:07:57 sync_lock: 
'GkJTcHFLl6YNc8M0QN7yYA0nWwhzHKjx7Vov15UStgsS1tgOISASG7bPjYf7NpYO' mode:1 
flags=1
CLVMD[6d152700]: Oct 11 13:07:57 sync_lock: returning lkid 6
CLVMD[6d152700]: Oct 11 13:07:57 getting initial lock for 
GkJTcHFLl6YNc8M0QN7yYA0nWwhzHKjx1OWkvGiyJGfz9u5Pcedbzj4hnT2Q6TY0
CLVMD[6d152700]: Oct 11 13:07:57 sync_lock: 
'GkJTcHFLl6YNc8M0QN7yYA0nWwhzHKjx1OWkvGiyJGfz9u5Pcedbzj4hnT2Q6TY0' mode:1 
flags=1
CLVMD[6d152700]: Oct 11 13:07:57 sync_lock: returning lkid 7
  Incorrect LVM locking library specified in lvm.conf, cluster operations may 
not work.
CLVMD[6d152700]: Oct 11 13:07:57 Sub thread ready for work.
CLVMD[6d152700]: Oct 11 13:07:57 LVM thread waiting for work
CLVMD[6d153740]: Oct 11 13:07:57 clvmd ready for work
CLVMD[6d153740]: Oct 11 13:07:57 Using timeout of 60 seconds
CLVMD[6d153740]: Oct 11 13:08:56 Got new connection on fd 11
CLVMD[6d153740]: Oct 11 13:08:56 Read on local socket 11, len = 29
CLVMD[6d153740]: Oct 11 13:08:56 check_all_clvmds_running
CLVMD[6d153740]: Oct 11 13:08:56 creating pipe, [12, 13]
CLVMD[6d153740]: Oct 11 13:08:56 Creating pre&post thread
CLVMD[6d153740]: Oct 11 13:08:56 Created pre&post thread, state = 0
CLVMD[6d131700]: Oct 11 13:08:56 in sub thread: client = 0x12d06d0
CLVMD[6d131700]: Oct 11 13:08:56 doing PRE command LOCK_VG 'P_#global' at 4 
(client=0x12d06d0)
CLVMD[6d131700]: Oct 11 13:08:56 sync_lock: 'P_#global' mode:4 flags=0
CLVMD[6d131700]: Oct 11 13:08:56 sync_lock: returning lkid 8
CLVMD[6d131700]: Oct 11 13:08:56 Writing status 0 down pipe 13
CLVMD[6d131700]: Oct 11 13:08:56 Waiting to do post command - state = 0
CLVMD[6d153740]: Oct 11 13:08:56 read on PIPE 12: 4 bytes: status: 0
CLVMD[6d153740]: Oct 11 13:08:56 background routine status was 0, 
sock_client=0x12d06d0
CLVMD[6d153740]: Oct 11 13:08:56 distribute command: XID = 0, flags=0x0 ()
CLVMD[6d153740]: Oct 11 13:08:56 add_to_lvmqueue: cmd=0x12d0a10. 
client=0x12d06d0, msg=0x12d02f0, len=29, csid=(nil), xid=0
CLVMD[6d153740]: Oct 11 13:08:56 Sending message to all cluster nodes
CLVMD[6d152700]: Oct 11 13:08:56 process_work_item: local
CLVMD[6d152700]: Oct 11 13:08:56 process_local_command: LOCK_VG (0x33) 
msg=0x12d0a50, msglen =29, client=0x12d06d0
CLVMD[6d152700]: Oct 11 13:08:56 do_lock_vg: resource 'P_#global', cmd = 0x4 
LCK_VG (WRITE|VG), flags = 0x4 ( DMEVENTD_MONITOR ), critical_section = 0
CLVMD[6d152700]: Oct 11 13:08:56 Refreshing context
  Incorrect metadata area header checksum on /dev/sdd at offset 4096
CLVMD[6d152700]: Oct 11 13:08:56 Reply from node node-10: 0 bytes
CLVMD[6d152700]: Oct 11 13:08:56 Got 1 replies, expecting: 2
CLVMD[6d152700]: Oct 11 13:08:56 LVM thread waiting for work
CLVMD[6d153740]: Oct 11 13:08:56 Reply from node node-11: 0 bytes
CLVMD[6d153740]: Oct 11 13:08:56 Got 2 replies, expecting: 2
CLVMD[6d131700]: Oct 11 13:08:56 Got post command condition...
CLVMD[6d131700]: Oct 11 13:08:56 Waiting for next pre command
CLVMD[6d153740]: Oct 11 13:08:56 read on PIPE 12: 4 bytes: status: 0
CLVMD[6d153740]: Oct 11 13:08:56 background routine status was 0, 
sock_client=0x12d06d0
CLVMD[6d153740]: Oct 11 13:08:56 Send local reply
CLVMD[6d153740]: Oct 11 13:08:56 Read on local socket 11, len = 28
CLVMD[6d131700]: Oct 11 13:08:56 Got pre command condition...
CLVMD[6d131700]: Oct 11 13:08:56 doing PRE command LOCK_VG 'V_samba4' at 1 
(client=0x12d06d0)
CLVMD[6d131700]: Oct 11 13:08:56 sync_lock: 'V_samba4' mode:3 flags=0
CLVMD[6d131700]: Oct 11 13:08:56 sync_lock: returning lkid 9
CLVMD[6d131700]: Oct 11 13:08:56 Writing status 0 down pipe 13
CLVMD[6d131700]: Oct 11 13:08:56 Waiting to do post command - state = 0
CLVMD[6d153740]: Oct 11 13:08:56 read on PIPE 12: 4 bytes: status: 0
CLVMD[6d153740]: Oct 11 13:08:56 background routine status was 0, 
sock_client=0x12d06d0
CLVMD[6d153740]: Oct 11 13:08:56 distribute command: XID = 1, flags=0x1 (LOCAL)
CLVMD[6d153740]: Oct 11 13:08:56 add_to_lvmqueue: cmd=0x12d0a10. 
client=0x12d06d0, msg=0x12d02f0, len=28, csid=(nil), xid=1
CLVMD[6d152700]: Oct 11 13:08:56 process_work_item: local
CLVMD[6d152700]: Oct 11 13:08:56 process_local_command: LOCK_VG (0x33) 
msg=0x12d0a50, msglen =28, client=0x12d06d0
CLVMD[6d152700]: Oct 11 13:08:56 do_lock_vg: resource 'V_samba4', cmd = 0x1 
LCK_VG (READ|VG), flags = 0x4 ( DMEVENTD_MONITOR ), critical_section = 0
CLVMD[6d152700]: Oct 11 13:08:56 Invalidating cached metadata for VG samba4
...