[Linux-cluster] GNBD and CLVM and multipath oh my

[David Golden]

Hi,

I have a 4-node GFS setup on top of a CLVM volume across a 
14-shared-block-device SAN.   I'd like to try farming out devices 
at a block level from the four head nodes to an, ahem, larger 
number of worker nodes, to see how GFS works across said larger 
number of nodes, otherwise it's back to NFS-ing out from the head 
nodes, which works but, jeez, it's NFS.

(I can delete and recreate the GFS filesystem at this
stage, so no worries about journals count etc.)

Soo... question, in case anyone else has tried this:

Which is most likely to work? 


(a) gnbd export the logical volume block device from the 
four head nodes to the worker nodes, (maybe multipath
the four imported gnbds on each worker node), then run gfs on top.

or

(b) gnbd export the underlying block devices (maybe multipath
them all into 14 multipath devices on each node)
and run  CLVM across the larger number of nodes,
then run gfs.

I guess this comes down to  whether and how GFS and CLVM are 
coupled... Naively, I'd imagine both "should" work, but (b) 
would cause unnecessary locking overhead, so (a) would be 
preferable, so I'll try (a) first unless someone says 
"no, you fool, (a) won't work because XYZ" before I get around
to it tomorrow.

("maybe multipath" because I'm using a RHEL4oid and AFAIK wouldn't
be able to make a working multipath-tools without a custom kernel...)

Best Regards,

David Golden