[Linux-cluster] Nodes are not joining to the cluster

[Srija]

Hi all,

Here is the issue  with the cluster  describing below:

The cluster is built with 16 nodes. All rhel5.5   86_64 bit OS.
yesterday night  two  servers were rebooted and after that  these
two servers are not joining to the cluster.

I was not  the part of the team when it is built. and my knowledge regarding cluster is also little bit.

Here is the scenario:

   -  There is no quorum  disks.  But the person
      who has built the cluster he is telling he has executed the quorum 
      from command line, [ i am not sure  of that ]

  -  The errors  in the message log  are showing as

ccsd[24182]: Unable to connect to cluster infrastructure after 12060 seconds , it is a continuous error message in the log file

The cluster.conf  are as follows:

<?xml version="1.0"?>
<cluster alias="newenvt" config_version="21" name="newenvt">
<fence_daemon clean_start="0" post_fail_delay="0" post_join_delay="3"/>


<clusternodes>

<clusternode name="host-priv.domain.org" nodeid="1" votes="1">
        <fence><method name="1">
        <device name="ilo-hostr"/></method>
        </fence>
</clusternode>

................... [  all the other  nodes  ]...................

 </clusternodes>
<cman/>

<dlm plock_ownership="1" plock_rate_limit="0"/>

<gfs_controld plock_rate_limit="0"/>


<fencedevices>

        <fencedevice agent="fence_ilo" hostname="hostr" login="Admin" name="hostr" passwd="xxxxxx"/>  
 
         .............................[  all the fence devices for other  nodes ]................

</fencedevices>

<rm>

<failoverdomains/>

<resources/>


</rm></cluster>

It seems it is a very basic configuration. But at this stage more important
is, to attach the two servers  in the cluster environment.

If more information is needed , i will provide.

Any advice is  appreciated.

Thanks in advance