[Linux-cluster] Multipath or Oracle RAC ASM issue?

[Bryn M. Reeves]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Joel Becker wrote:
> 	This works as you expect.  The names in
> /dev/(mapper|mpath)/mpathX should be identical on each server.

It's often best to avoid the symlinks in /dev/mpath. Depending on your
version of udev they may get seriously out of sync with device creation,
meaning that the symlinks might not exist at the point you try to use
them. This is especially a problem with older udevs and for e.g.
mounting file systems using /dev/mpath/mpathN entries in fstab.

> different.  After they are created, the udev subsystem uses the
> information in /var/lib/multipath/bindings to map the mpathX name to the
> appropriate dm-X device.  This ensures that the mpathX names are
> identical on each system.

multipath/multipathd manage the mpathX names directly (creating them via
libdevmapper). It's udev that adds the symlinks (possible at some later
time).

>> Q2 :
>> How do we make it such that the dm-x  devices are accessing the same
>> SAN LUNs across the servers?  I believe they are not the same based
>> on the observations of the disk spaces associated with each of the dm-x
>> shown in /proc/partition
> 
> 	You don't.  You don't care.  If you want a /dev name, you use
> the mpathX name that you know maps correctly.  Since you are using
> ASMLib, you don't worry about that either.  The LANDx names are read by
> ASMLib to ensure ASM sees the correct devices no matter what /dev name
> they have.

It is important here to make sure that the aliases are synchronised. You
can do this by syncing the bindings file between the hosts or by using
explicit alias { /*...*/ } entries in multipath.conf.

Watch out if you have /var as a separate file system though - you can
get situations where mpathN names change unpredictably as the system
boots up (initial multipath run happens before /var is mounted, causing
multipath to think there is no bindings file. Later when /var is mounted
multipath or multipathd will try to change the mappings to honour the
bindings file causing devices to flip around). This can lead to data
loss/corruption in the the administrator cannot be completely sure the
device he/she thinks mpathN corresponds to really maps to the intended
device.

Recent versions of multipath-tools have a "bindings_file" option that
allows you to specify an alternate location to avoid this problem (e.g.
/etc/multipath-bindings).

>> We logged a call to Oracle who responded :
>> If we are using device mapper and ASMLib then, we need to use disks from
>> /dev/dm-* disks
>> instead of disks from  /dev/mapper/mpath*
> 
> 	That's not the issue, and I'm sorry Oracle support gave you the
> wrong answer.  You can createdisk against any name that's correct
> (/dev/dm-X, /dev/mapper/mpathX, /dev/mpath/mpathX).  scandisks doesn't
> even need a name - it just needs the correct order in your case.

Yeah, that's unfortunate. The dm-N nodes are really internal
device-mapper names and the general advice is to always prefer the
/dev/mapper names. Recent distributions tend to have udev rules that
ignore these devices & avoid creating the /dev/dm-N nodes completely.

Regards,
Bryn.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iEYEARECAAYFAkhZnbsACgkQ6YSQoMYUY94fhwCgkrZPrT63kU4CDeipGdsscNbP
casAn2ZYi08ZCycIkQAe+5DVvDlcxrFv
=yIMT
-----END PGP SIGNATURE-----