[Linux-cluster] clvmd, can't see shared storage on node1 of 2-node cluster (long)
linux-cluster at merctech.com
linux-cluster at merctech.com
Mon Jan 28 23:14:09 UTC 2008
I'm having trouble with a new two-node CentOS5 cluster (kernel 2.6.23). The clvm
daemon doesn't run correctly on one node, and that node cannot see LVM objects,
while it can connect to the underlying storage.
The cluster has shared SAN storage with 5 LUNs presented to the servers.
Multiple HBAs and storage processes mean there are 4 paths to each LUN.
DM-Multipath is used to create virtual devices for LUN access. Each server sees
the LUNs at the SCSI level, and can see disk partitions via fdisk, and can see
the same LUNs via multipath
The host-level storage configuration was done on node2, with text files being
edited simultaneously on both nodes or copied from node2 to node1.
### Error Condition ###
The clvmd daemon does not run correctly on node1 of the two-node cluster.
The error messages are:
connect() failed on local socket: Permission denied
WARNING: Falling back to local file-based locking.
Volume Groups with the clustered attribute will be inaccessible.
At this point, node1 cannot access any shared objects (ie., pvdisplay fails to
show details about the physical volumes, etc.).
The daemon consistently starts without error on node2, whether node1 is
running or not.
The daemon consistently fails to start on node1, whether node2 is running or
not.
I've rebooted each node...the condition remains the same--only node2 can
successfully start clvmd and access the LVM volumes.
The /var/log/messages entry on both nodes is identical:
clvmd: Cluster LVM daemon started - connected to CMAN
but on node1 the process never gets out of the " clvmd -T20" state.
There are no issues with SELinux blocking any lvm actions.
I'd really appreciate any suggestions about debugging this problem.
Extensive notes and command output are given
below.
Thanks,
Mark
---------------------------------------------------------------------------------
#### Configuration Procedures ###
The following steps were run on each node:
dm-multipath installed, confirmed that kernel modules are loaded
setup /etc/multipath.conf (aliasing WWIDs to logical names,
blacklisting WWIDs of internal drive)
stop multipathd
remove any multipath device entries with
multipath -F
Create /etc/lvm/lvm.conf (filtering internal drive, filtering /dev/sda
devices
The following commands were run _only_ on node2:
restart multipath, recreating the devices based on /etc/multipath.conf
copy the /var/lib/multipath/bindings file from node2 to node1
Create Physical Devices for LVM, as in:
pvcreate -M2 -v /dev/mpath/home
Create Volume Groups, as in:
vgcreate -c y home_vg /dev/mpath/home
Create Logical Volumes, as in:
lvcreate -l 100%VG -n archive archive_vg
Once the volumes are created, the Major/Minor block numbers were set
to be persistent in order to help with NFS load balancing and
fail-over across cluster nodes using lvchange, as in:
lvchange --persistent y --major 253 --minor 8 archive_vg/archive
Partition the Logical Volumes
fdisk was used to set the filesystem offset to 128 blocks
(64KB) to avoid boundary crossing on the EMC array
Filesystems
All paritions are formatted as gfs type filesystems as in:
gfs_mkfs -j 4 -O -p lock_dlm -t sbia-infr:archive /dev/archive_vg/archive
### Component Versions ###
clvmd version:
Cluster LVM daemon version: 2.02.26-RHEL5 (2007-06-18)
Protocol version: 0.2.1
pv* versions:
Library version: 1.02.20 (2007-06-15)
Driver version: 4.11.0
rgmanager:
Version : 2.0.31 Vendor: CentOS
Release : 1.el5.centos Build Date: Mon Nov 12 01:13:08 2007
#################################### Configuration ################################
The /etc/lvm/lvm.conf files are identical on both nodes. The locking value is
set to:
locking_type = 3
### DM Multipath Details ###
Both nodes have identical multipath configurations (/etc/multipath.conf,
/dev/mpath/*, /dev/dm-* are identical). Both nodes show the same devices from
"fdisk -l":
[root at sbia-infr2 mpath]# fdisk -l /dev/mpath/*
Disk /dev/mpath/archive: 1407.4 GB, 1407450152960 bytes
255 heads, 63 sectors/track, 171112 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/mpath/archive doesn't contain a valid partition table
Disk /dev/mpath/cluster_shared: 288.1 GB, 288161071104 bytes
255 heads, 63 sectors/track, 35033 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/mpath/cluster_shared doesn't contain a valid partition table
Disk /dev/mpath/comp_space: 2017.1 GB, 2017127497728 bytes
255 heads, 63 sectors/track, 245235 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/mpath/comp_space doesn't contain a valid partition table
Disk /dev/mpath/home: 1152.6 GB, 1152644284416 bytes
255 heads, 63 sectors/track, 140134 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/mpath/home doesn't contain a valid partition table
Disk /dev/mpath/sbiaprj: 2017.1 GB, 2017127497728 bytes
255 heads, 63 sectors/track, 245235 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/mpath/sbiaprj doesn't contain a valid partition table
### Example of clvmd, pvdisplay errors; group_tool, clustat, "clvmd -d" output ###
================ Node1 ====================================================
[root at sbia-infr1 lvm]# /etc/init.d/clvmd start
Starting clvmd: [ OK ]
Activating VGs: Logging initialised at Mon Jan 28 16:01:41 2008
Set umask to 0077
connect() failed on local socket: Permission denied
WARNING: Falling back to local file-based locking.
Volume Groups with the clustered attribute will be inaccessible.
Finding all volume groups
Finding volume group "home_vg"
Skipping clustered volume group home_vg
Finding volume group "cluster_shared_vg"
Skipping clustered volume group cluster_shared_vg
Finding volume group "sbiaprj_vg"
Skipping clustered volume group sbiaprj_vg
Finding volume group "comp_space_vg"
Skipping clustered volume group comp_space_vg
Finding volume group "archive_vg"
Skipping clustered volume group archive_vg
Finding volume group "VolGroup00"
2 logical volume(s) in volume group "VolGroup00" already active
2 existing logical volume(s) in volume group "VolGroup00" monitored
Found volume group "VolGroup00"
Found volume group "VolGroup00"
Activated logical volumes in volume group "VolGroup00"
2 logical volume(s) in volume group "VolGroup00" now active
Wiping internal VG cache
[root at sbia-infr1]# group_tool -v
type level name id state node id local_done
fence 0 default 00010002 JOIN_START_WAIT 1 100020001 1
[1 2]
dlm 1 rgmanager 00030001 none
[1 2]
dlm 1 clvmd 00050001 none
[root at sbia-infr1 mpath]# clustat
Member Status: Quorate
Member Name ID Status
------ ---- ---- ------
sbia-infr2-admin.uphs.upenn.edu 1 Online, rgmanager
sbia-infr1-admin.uphs.upenn.edu 2 Online, Local, rgmanager
Service Name Owner (Last) State
------- ---- ----- ------ -----
service:vweb (none) stopped
[root at sbia-infr1]# clvmd -d
[clvmd PRODUCES THE FOLLOWING DEBUGGING OUTPUT ON STARTUP]
CLVMD[4f71c4c0]: Jan 28 17:03:46 CLVMD started
CLVMD[4f71c4c0]: Jan 28 17:03:46 Connected to CMAN
CLVMD[4f71c4c0]: Jan 28 17:03:46 CMAN initialisation complete
CLVMD[4f71c4c0]: Jan 28 17:03:47 DLM initialisation complete
CLVMD[4f71c4c0]: Jan 28 17:03:47 Cluster ready, doing some more initialisation
CLVMD[4f71c4c0]: Jan 28 17:03:47 starting LVM thread
CLVMD[4f71c4c0]: Jan 28 17:03:47 clvmd ready for work
CLVMD[4f71c4c0]: Jan 28 17:03:47 Using timeout of 60 seconds
CLVMD[41001940]: Jan 28 17:03:47 LVM thread function started
Logging initialised at Mon Jan 28 17:03:47 2008
Set umask to 0077
CLVMD[41001940]: Jan 28 17:03:47 LVM thread waiting for work
CLVMD[4f71c4c0]: Jan 28 17:04:49 Got port closed message, removing node sbia-infr2-admin.uphs.upenn.edu
CLVMD[4f71c4c0]: Jan 28 17:05:04 add_to_lvmqueue: cmd=0x884bc0. client=0x65c8a0, msg=0x7fff5b3a91ac, len=30, csid=0x7fff5b3a90f4, xid=0
CLVMD[41001940]: Jan 28 17:05:04 process_work_item: remote
CLVMD[41001940]: Jan 28 17:05:04 process_remote_command 2 for clientid 0x0 XID 128 on node sbia-infr2-admin.uphs.upenn.edu
CLVMD[41001940]: Jan 28 17:05:04 Remote node sbia-infr2-admin.uphs.upenn.edu is version 0.2.1
CLVMD[41001940]: Jan 28 17:05:04 Added new node 1 to updown list
CLVMD[41001940]: Jan 28 17:05:04 LVM thread waiting for work
[AT THIS POINT, THERE IS NO OUTPUT FROM clvmd IN RESPONSE TO
"pvdisplay"]
=============================================================================
================ Node2 ====================================================
[root at sbia-infr2 lvm]# /etc/init.d/clvmd start
Starting clvmd: [ OK ]
Activating VGs: Logging initialised at Mon Jan 28 16:01:41 2008
Set umask to 0077
Finding all volume groups
Finding volume group "home_vg"
Activated logical volumes in volume group "home_vg"
1 logical volume(s) in volume group "home_vg" now active
Finding volume group "cluster_shared_vg"
Activated logical volumes in volume group "cluster_shared_vg"
1 logical volume(s) in volume group "cluster_shared_vg" now active
Finding volume group "sbiaprj_vg"
Activated logical volumes in volume group "sbiaprj_vg"
1 logical volume(s) in volume group "sbiaprj_vg" now active
Finding volume group "comp_space_vg"
Activated logical volumes in volume group "comp_space_vg"
1 logical volume(s) in volume group "comp_space_vg" now active
Finding volume group "archive_vg"
Activated logical volumes in volume group "archive_vg"
1 logical volume(s) in volume group "archive_vg" now active
Finding volume group "VolGroup00"
2 logical volume(s) in volume group "VolGroup00" already active
2 existing logical volume(s) in volume group "VolGroup00" monitored
Activated logical volumes in volume group "VolGroup00"
2 logical volume(s) in volume group "VolGroup00" now active
Wiping internal VG cache
[root at sbia-infr2 lvm]# group_tool -v
type level name id state node id local_done
fence 0 default 00010001 JOIN_START_WAIT 2 200020001 1
[1 2]
dlm 1 rgmanager 00030001 none
[1 2]
dlm 1 clvmd 00050001 none
[1 2]
[root at sbia-infr2 mpath]# clustat
Member Status: Quorate
Member Name ID Status
------ ---- ---- ------
sbia-infr2-admin.uphs.upenn.edu 1 Online, Local, rgmanager
sbia-infr1-admin.uphs.upenn.edu 2 Online, rgmanager
Service Name Owner (Last) State
------- ---- ----- ------ -----
service:vweb (none) stopped
[root at sbia-infr2]# clvmd -d
[ISSUING "pvdisplay" IN ANOTHER WINDOW ON THE SAME SERVER
PRODUCES THE FOLLOWING DEBUGGING OUTPUT FROM clvmd]
CLVMD[3b52c4c0]: Jan 28 17:05:03 CLVMD started
CLVMD[3b52c4c0]: Jan 28 17:05:03 Connected to CMAN
CLVMD[3b52c4c0]: Jan 28 17:05:03 CMAN initialisation complete
CLVMD[3b52c4c0]: Jan 28 17:05:04 DLM initialisation complete
CLVMD[3b52c4c0]: Jan 28 17:05:04 Cluster ready, doing some more initialisation
CLVMD[3b52c4c0]: Jan 28 17:05:04 starting LVM thread
CLVMD[41001940]: Jan 28 17:05:04 LVM thread function started
CLVMD[3b52c4c0]: Jan 28 17:05:04 clvmd ready for work
CLVMD[3b52c4c0]: Jan 28 17:05:04 Using timeout of 60 seconds
Logging initialised at Mon Jan 28 17:05:04 2008
Set umask to 0077
File descriptor 5 left open
Logging initialised at Mon Jan 28 17:05:04 2008
Finding all logical volumes
Wiping internal VG cache
CLVMD[41001940]: Jan 28 17:05:04 LVM thread waiting for work
CLVMD[3b52c4c0]: Jan 28 17:08:48 Got new connection on fd 9
CLVMD[3b52c4c0]: Jan 28 17:08:48 Read on local socket 9, len = 30
CLVMD[3b52c4c0]: Jan 28 17:08:48 creating pipe, [10, 11]
CLVMD[3b52c4c0]: Jan 28 17:08:48 Creating pre&post thread
CLVMD[3b52c4c0]: Jan 28 17:08:48 Created pre&post thread, state = 0
CLVMD[41802940]: Jan 28 17:08:48 in sub thread: client = 0x884bc0
CLVMD[41802940]: Jan 28 17:08:48 Sub thread ready for work.
CLVMD[41802940]: Jan 28 17:08:48 doing PRE command LOCK_VG 'V_home_vg' at 1 (client=0x884bc0)
CLVMD[41802940]: Jan 28 17:08:48 sync_lock: 'V_home_vg' mode:3 flags=0
CLVMD[41802940]: Jan 28 17:08:48 sync_lock: returning lkid 430001
CLVMD[41802940]: Jan 28 17:08:48 Writing status 0 down pipe 11
CLVMD[41802940]: Jan 28 17:08:48 Waiting to do post command - state = 0
CLVMD[3b52c4c0]: Jan 28 17:08:48 read on PIPE 10: 4 bytes: status: 0
CLVMD[3b52c4c0]: Jan 28 17:08:48 background routine status was 0, sock_client=0x884bc0
CLVMD[3b52c4c0]: Jan 28 17:08:48 distribute command: XID = 0
CLVMD[3b52c4c0]: Jan 28 17:08:48 add_to_lvmqueue: cmd=0x884ff0. client=0x884bc0, msg=0x8785d0, len=30, csid=(nil), xid=0
CLVMD[41001940]: Jan 28 17:08:48 process_work_item: local
CLVMD[41001940]: Jan 28 17:08:48 process_local_command: msg=0x885030, msglen =30, client=0x884bc0
[HUNDREDS OF LINES OF OUTPUT DELETED...]
=============================================================================
-----
Mark Bergman
http://wwwkeys.pgp.net:11371/pks/lookup?op=get&search=bergman%40merctech.com
More information about the Linux-cluster
mailing list