[Linux-cluster] Re: If I have 5 GNBD server?

[brianu]

Hello all,

I tried creating a md device from the GNBD's and I was able to create one, I
created a raid 1 device (however it is still going thru its sync process(yet
there is no difference the volume is from shared storage)). The problem is I
cannot mount the device as shown below

Aug 31 14:46:27 dell-1650-31 kernel: GFS: Trying to join cluster "lock_dlm",
"sansilvercash:gfs2"
Aug 31 14:46:27 dell-1650-31 kernel: lock_dlm: fence domain not found; check
fenced Aug 31 14:46:27 dell-1650-31 kernel: GFS: can't mount proto =
lock_dlm, table = sansilvercash:gfs2, hostdata = Aug 31 14:46:39
dell-1650-31 hal.hotplug[5236]: timout(10000 ms) waiting for
/block/diapered_dm-1 Aug 31 14:50:01 dell-1650-31 crond(pam_unix)[7698]:
session opened for user root by (uid=0) Aug 31 14:50:01 dell-1650-31
crond(pam_unix)[7698]: session closed for user root.

Yet I can mount the device by gnbds individually.

PVSCAN shows:

[root at dell-1650-31 ~]# pvscan
  PV /dev/sda3   VG VolGroupData   lvm2 [31.81 GB / 32.00 MB free]
  PV /dev/md0    VG space          lvm2 [1.07 TB / 0    free]
  Total: 2 [1.10 TB] / in use: 2 [1.10 TB] / in no VG: 0 [0   ]
[root at dell-1650-31 ~]#


Any clues on why I wouldn't be able to mount the device? This is the same
behavior when I tried to mount a multipathed device ( which I may have
configured right or wrong)

-Brian
-----Original Message-----
From: brianu [mailto:brianu at silvercash.com] 
Sent: Wednesday, August 31, 2005 10:26 AM
To: linux-cluster at redhat.com
Cc: 'brianu'
Subject: FW: [Linux-cluster] Re: If I have 5 GNBD server?

Ok, so I tried using heartbeat to create a virtual ip that floats between
the gnbd servers it worked out okay I can actually mount under that virtual
gnbd server ip. And If I wanted to manually fail out one gnbd server for
another no prob if all was done cleany, i.e. ( vgchange -aln, gnbd_import
-R) then would remount by gnbd_import -i gnbd_vip:

Which would give the unique gnbd in /dev/gnbd, then I'd run vgchange -aly,
would would bring it in lvm and device mapper.

However a manual failover test hangs with "vgchange -aln" as the old device
"unique gnbd" is still attempted to be accessed, also kill the process with
a killall or a kill -11, still doesn't cleanly allow clmvd to return to a
clean state.


As for Multipath as  Ben has wrote

> If the gnbds are exported uncached (the default), the client will fail
back IO
> if it can no longer talk to the server after a specified timeout.  However
> the userspace tools for dm-multipath are still to SCSI-centric to allow
you
> to run on top of gnbd.  You can manually run dm-setup commands to build
> the appropriate multipath map, scan the map to check if a path has failed,
> remove the failed gnbd from the map (so the device can close and gnbd can
> start trying to reconnect), and them manually add the gnbd device back
into
> the map when it has reconnected. That's pretty much all the dm-multipath
> userspace tools do.  Someone could even write a pretty simple daemon that
did
> this, and become the personal hero of many people on this list.

> The only problem is that if you manually execute the commands, or write
the
> daemon in bash or some other scripting language, you can run into a memory
> deadlock. If you are in a very low memory situation, and you need to
complete
> gnbd IO requests to free up memory, the daemon can't allocate any memory
in
> doing it's job.
& 
> If you have the gnbd exported in caching mode, each server will maintain
it's
> own cache, So if you write a block to one server, and then the server
crashes,
> when you read the block from the second server, if it was already cached
> before the read, you will get invalid data, so that won't work. If you
> set the gnbd to uncached mode, the client will fail the IO back, and
something
> (a multipath driver) needs to be there to reissue the request.

> -Ben

I have tried to get the dm-multipath setup working correctly but had little
success I had started a earlier thread on it and didn't get any response,

My test was based on 
https://www.redhat.com/archives/linux-cluster/2005-April/msg00062.html

I posted my issues here.

https://www.redhat.com/archives/linux-cluster/2005-August/msg00080.html

The initial command used was 
echo "0 1146990592  multipath 0 0 1 1 round-robin 0 2 1 251:1 1000 251:3
1000 " | dmsetup create dm-1

(251:1 & 251:3 are the major:minor ids of the gnbds obtained from the
command cat /proc/partitions)
(1146990592 -> I believe is the size of the block device.) 
This resulted in a block device of which I still could not mount, I tried
multipath -0ll (after installing multipath and create a rudimentary
multipath.conf) and the result was

dm-1

[size=546 GB][features="0"][hwhandler="0"] \_ round-robin 0 [active][first]
  \_ 0:0:0:0      251:0   [undef ][active]
\_ round-robin 0 [enabled]
  \_ 0:0:0:0      251:4   [undef ][active]

"notice that the size was 1/2 the actual size!?! (I have no idea what this
means "somebody enlighten me, please!)

When I attempted to mount
 
[root dell-1650-31 ~]# mount -t gfs /dev/mapper/dm-1 /mnt/gfs1

mount: /dev/dm-1: can't read superblock

This was tried previously off a multipathed device in  which dmsetup status
gives the output below:

dm-1: 0 1146990592 multipath 1 0 0 2 1 A 0 1 0 251:1 A 0 E 0 1 0 251:3 A 0

dmsetup deps gives
dm-1: 2 dependencies    : (251, 3) (251, 1)

and dmsetup info gives
Name:              dm-1
State:             ACTIVE
Tables present:    LIVE
Open count:        0
Event number:      0
Major, minor:      253, 1
Number of targets: 1

I have managed to get a nfs/gnbd failover type scenario working, in which
gnbd_servers export the shared storage via nfs and the clients mount via a
heartbeat VIP. I then created a script which I will rewrite into into a mini
daemon soon, that checks status of the servers then when the ip is token
over stops apache, unmounts nfs via the "umount -l -t nfs $mountpoint" then
"mount $mountpoint" & starts apache again. I have tested it and it works (
by checking for stale handles and remounting cleanly) Some of the same
principles can go into one for gnbd. But my bump right now is LVM.


Can GNBD be used without LVM? Or does anyone know how to enable failover
correctly on dm-multipath?


Any help would be appreciated.

Brian Urrutia 


-----Original Message-----
From: brian urrutia [mailto:brianu at mail.silvercash.com] 
Sent: Monday, August 29, 2005 12:56 AM
To: linux-cluster at redhat.com
Cc: mikore.li at gmail.com; brianu
Subject: Re: [Linux-cluster] Re: If I have 5 GNBD server?

> > If using LVM to make a volume of imported gnbds is not the answer for
> > redundancy can anyone suggest a method that is? Im not opposed to using
any
> > other resource of cluster or GFS but I would really like to implement a
> > redundant solution, ( gnbd, gulm, etc.). 
> > 
> Hi, Brianu, maybe LVM + md + gnbd should be one of the solution for
> redundancy, for example, you have 2 gnbd servers, each one exports 1
> disk. Then, the steps should be:
> 1. create a RAID-1  /dev/md0 on GFS client with imported 2 gnbd block
devices.
> 2. use LVM  create /dev/vg0 on top of them.
> 3. mkfs_gfs on /dev/vg0.
> I haven't tried this configuration, theoretically, it should work.
> 
> Thanks,
> 
> Michael

I will look into trying a md & lvm combo, as far as keepalived or
rgmanager to failover an ip, i havent seen a clear example on how to use
rgmanager, but i have tried heartbeat (linux HA) to failover the ip, and
the problem is that the gnbd clients still seem to lock on the former
server regardless of that the ip has failed over to another ip ( and
continuly try and reconnect as Fajar had mentioned).


The shared storage I have is a HP MSA 100 SAN
It might be a config error on my part as far as rgmanager is concerned i
will have to post my cluster.conf tommorrow.

-Brian