[Linux-cluster] Using GFS without a network?

[Steve Wilcox]

On Wed, 2005-09-07 at 00:57 +0200, Andreas Brosche wrote:

> > - Multi-initator SCSI buses do not work with GFS in any meaningful way,
> > regardless of what the host controller is.
> > Ex: Two machines with different SCSI IDs on their initiator connected to
> > the same physical SCSI bus.
> 
> Hmm... don't laugh at me, but in fact that's what we're about to set up.
> 
> I've read in Red Hat's docs that it is "not supported" because of 
> performance issues. Multi-initiator buses should comply to SCSI 
> standards, and any SCSI-compliant disk should be able to communicate 
> with the correct controller, if I've interpreted the specs correctly. Of 
> course, you get arbitrary results when using non-compliant hardware... 
> What are other issues with multi-initiator buses, other than performance 
> loss?

I set up a small 2 node cluster this way a while back, just as a testbed
for myself.  Much as I suspected, it was severely unstable because of
the storage configuration, even occasionally causing both nodes to crash
when one was rebooted due to SCSI bus resets.  I tore it down and
rebuilt it several times, configuring it as a simple failover cluster
with RHEL3 and RHEL4, a GFS cluster under RHEL4 and Fedora4, and as an
openSSI cluster using Fedora3.  All tested configurations were equally
crash-happy due to the bus resets.  

My configuration consisted of a couple of old Compaq deskpro PC's, each
with a single ended Symbiosis card (set to different SCSI ID's
obviously) and an external DEC BA360 jbod shelf with 6 drives.  The bus
resets might be mitigated somewhat by using HVD SCSI and Y-cables with
external terminators, but from my previous experience with other
clusters that used this technique (DEC ASE and HP-ux service guard), bus
resets will always be a thorn in your side without a separate,
independent raid controller to act as a go-between.  Calling these
configurations simply "not supported" is an understatement - this type
of config is guaranteed trouble.  I'd never set up a cluster this way
unless I'm the only one using it, and only then if I don't care one
little bit about crashes and data corruption.  My two cents.

-steve