[Linux-cluster] RedHat RHEL 5U4 NFS Cluster nodes randomly reboot

[Volker Dormeyer]

Hi,

On Tue, Jul 13, 2010 at 05:39:30PM -0500,
Bennie Thomas <Bennie_R_Thomas at raytheon.com> wrote:
> I think I may have found my problem. However, I am not sure how to fix it.
> 
> I noticed that node one shows the quorum disk to be /dev/dm-2 and
> node 2 shows the quorum disk to be /dev/dm-3

How do you configured the cluster to use the Quorum? By Label or by Device?
If you used the device settings and specified a dm-device in cluster.conf,
this would be your problem.

If you used the Quorum-label, it shouldn't. The underlying dm-device-name
doesn't matter as long as you do not specify it explicitly. The devices will
be mapped to the appropriate dm-device, if you use /dev/mapper/... devices.
Hence, I don't beleive this to be your problem.

Do you have cman killed messages in the log of the remaining node before it is
being fenced? May be you can post the appropriate part of the log and the
config.

It might be helpful to run a debug-log for qdiskd - just to see whether your
qdisk is the root cause. Could you add the options log_level="7" and
log_facility="local3" to your qdisk configuration part? You need to setup
an appropriate entry in /etc/syslog.conf then, as well. Qdiskd logs some debug
messages, if it is not able to access the qdisk, etc.

> How can I change these ?

However, I think your Quorum Device is located on a logical volume. Hence, it
makes sense to force LVM to use descriptive device names by setting the device
filter. I. e.:

preferred_names = [ "^/dev/mapper/mpath", "^/dev/[hs]d" ]
filter = [ "a|/dev/mapper/mpath.*|", "a|/dev/cciss/.*|", "r|.*|" ]

Afterwards, perfrom "pvscan" and LVM should be using the descriptive names.
"pvs" should now list "/dev/mapper/mpath*" devices. Of course, you have to
change the settings if you do not use multipathing. You local storage
controller should be listed in the filter, as well...

The filter will not solve your problem, but makes the configuration more
transparent.

If your quorum device is located on a self-made dm-device instead, just create
it with a unique name. 

Regards,
Volker