[Linux-cluster] caching of san devices....

[Florian Haas]

On Tue, Apr 3, 2012 at 3:14 PM, Alan Brown <ajb2 at mssl.ucl.ac.uk> wrote:
>
> Real Dumb Question[tm] time....
>
> Has anyone tried putting bcache/flashcache in front of shared storage in a
> GFS2 cluster (on each node, of course)

I can't talk about bcache but have worked with flashcache a bit, and
there's a presentation of mine on how to use it in clustering at
http://www.hastexo.com/resources/presentations/storage-replication-high-performance-high-availability-environments
(which is all about Pacemaker, though).

But for GFS2 specifically:

> Did it work?

It won't.

> Should it work?

No.

> Is it safe?

No. There's no cluster awareness the way you envision it, and there's
no way to do multi-master replication of the flashcache cache device,
which you would need.

> Are there ways of making it safe?

Implement the above, and it might be. (You don't want to.)

> Am I mad for thinking about it?

Ahum, well, now that you mention it... ;)

> Rationale:
>
> Spinning disks are slow to seek, large arrays even more so.
>
> As soon as there's a significant load on our GFS2 cluster the random io
> limitations of the SAN hardware become the single most important factor
> limiting performance.
>
> Only "so much" ram can be installed in any hardware to increase page and
> dentry caching before physical limits are hit.
>
> SSD SAN arrays are hideously expensive and can't always be justified to "the
> powers that be".
>
> Universities are always tightly funded, but there are many other entities
> facing similar problems.

I think you've got two possibilities:

1. Stick SSD based caching into your SAN. Google for CacheCade or
MaxCache for some vendor implementations.
2. Consider ditching your GFS2 for SSD based GlusterFS replication.

I realize option 2 may get me booed off the list, and I know nothing
about your requirements other than what you posted here, but if you
just want something that is writable from all nodes and frees you from
your SAN, then that might be a possibility.

Cheers,
Florian

-- 
Need help with High Availability?
http://www.hastexo.com/now