[Linux-cluster] Using GFS without a network?

[Bob Marcan]

Steve Wilcox wrote:
> On Wed, 2005-09-07 at 19:43 +1000, Keith Hopkins wrote:
> 
>>Steve Wilcox wrote:
>>
>>>On Tue, 2005-09-06 at 20:06 -0400, Steve Wilcox wrote:
>>>
>>>
>>>>On Wed, 2005-09-07 at 00:57 +0200, Andreas Brosche wrote:
>>>>
>>>>
>>>>
>>>>>>- Multi-initator SCSI buses do not work with GFS in any meaningful way,
>>>>>>regardless of what the host controller is.
>>>>>>Ex: Two machines with different SCSI IDs on their initiator connected to
>>>>>>the same physical SCSI bus.
>>>>>
>>>>>Hmm... don't laugh at me, but in fact that's what we're about to set up.
>>>>>
>>>>>I've read in Red Hat's docs that it is "not supported" because of 
>>>>>performance issues. Multi-initiator buses should comply to SCSI 
>>>>>standards, and any SCSI-compliant disk should be able to communicate 
>>>>>with the correct controller, if I've interpreted the specs correctly. Of 
>>>>>course, you get arbitrary results when using non-compliant hardware... 
>>>>>What are other issues with multi-initiator buses, other than performance 
>>>>>loss?
>>>>
>>>>I set up a small 2 node cluster this way a while back, just as a testbed
>>>>for myself.  Much as I suspected, it was severely unstable because of
>>>>the storage configuration, even occasionally causing both nodes to crash
>>>>when one was rebooted due to SCSI bus resets.  I tore it down and
>>>>rebuilt it several times, configuring it as a simple failover cluster
>>>>with RHEL3 and RHEL4, a GFS cluster under RHEL4 and Fedora4, and as an
>>>>openSSI cluster using Fedora3.  All tested configurations were equally
>>>>crash-happy due to the bus resets.  
>>>>
>>>>My configuration consisted of a couple of old Compaq deskpro PC's, each
>>>>with a single ended Symbiosis card (set to different SCSI ID's
>>>>obviously) and an external DEC BA360 jbod shelf with 6 drives.  The bus
>>>>resets might be mitigated somewhat by using HVD SCSI and Y-cables with
>>>>external terminators, but from my previous experience with other
>>>>clusters that used this technique (DEC ASE and HP-ux service guard), bus
>>>>resets will always be a thorn in your side without a separate,
>>>>independent raid controller to act as a go-between.  Calling these
>>>>configurations simply "not supported" is an understatement - this type
>>>>of config is guaranteed trouble.  I'd never set up a cluster this way
>>>>unless I'm the only one using it, and only then if I don't care one
>>>>little bit about crashes and data corruption.  My two cents.
>>>>
>>>>-steve
>>>
>>>
>>>
>>>Small clarification - Although clusters from DEC, HP, and even
>>>DigiComWho?Paq's TruCluster can be made to work (sort of) on multi-
>>>initiator SCSI busses, IIRC it was never a supported option for any of
>>>them (much like RedHat's offering).  I doubt any sane company would ever
>>>support that type of config.
>>>
>>>-steve   
>>>
>>
>>HP-UX ServiceGuard words well with multi-initiator SCSI configurations, and is fully supported by HP.  It is sold that way for small 2-4 node clusters when cost is an issue, although FC has become a big favorite (um...money maker) in recent years.  Yes, SCSI bus resets are a pain, but they are handled by HP-UX, not ServiceGuard.
>>
>>--Keith
> 
> 
> Hmmm...   Are you sure you're thinking of a multi-initiator _bus_ and
> not something like an external SCSI array (i.e. nike arrays or some such
> thing)?  I know that multi-port SCSI hubs are available, and more than
> one HBA per node is obviously supported for multipathing, but generally
> any multi-initiator SCSI setup will be talking to an external raid
> array, not a simple SCSI bus, and even then bus resets can cause grief.
> Admittedly, I'm much more familiar with the Alpha server side of things

==========================> should be unfamiliar

> (multi-initiator buses were definitely never supported under DEC unix /
> Tru64) , so I could be wrong about HP-ux.  I just can't imagine that a
> multi-initiator bus wouldn't be a nightmare.   
> 
> -steve
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> http://www.redhat.com/mailman/listinfo/linux-cluster

In the past i was using the SCSI cluster on OpenVMS(AXP,VAX) and Tru64.
At home i have 2 DS10 with memmory channel, shared SCSI Tru64 cluster.
Memmory channel was prerequisite in early days, now you can use ethernet 
as CI.
I still have some customers using the SCSI cluster on Tru64.
Two of this are banks, running this configuration a few years.
Without any problems. Using host based shadowing.
Tru64 has single point of failure in this configuration.
Quorum disk can't be shadowed.
OpenVMS doesn't have this limitation.

It is supported.

OpenVMS 
http://h71000.www7.hp.com/doc/82FINAL/6318/6318pro_002.html#interc_sys_table

Tru64
http://h30097.www3.hp.com/docs/base_doc/DOCUMENTATION/V51B_HTML/ARHGWETE/CHNTRXXX.HTM#sec-generic-cluster
...

Best regards, Bob

-- 
  Bob Marcan, Consultant                mailto:bob.marcan at snt.si
  S&T Hermes Plus d.d.                  tel:   +386 (1) 5895-300
  Slandrova ul. 2                       fax:   +386 (1) 5895-202
  1231 Ljubljana - Crnuce, Slovenia     url:   http://www.snt.si