[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [rhelv5-list] support for very large volumes 30 TB -- filesystems, architecture

On May 22, 2008, at 8:36 PM, John Summerfield wrote:

Terry Davis wrote:
I am going to be implementing an NFS cluster on RHEL5 x86_64 that I would like to serve up a single very large volume (one volume for application reasons). Right now my SAN max volume capability is 15 TB. So, I'd like to use LVM and tie several physical volumes together into a sigle logical volume. As stated, the cluster will be serving this volume up via NFS and the data is random files but I estimate the number of files to be around 70
million per 10 TB of data.
1) What file system should I use? Ext3 appears to have a 4k block size limitation which puts my max size at 8 TB. Reiserfs doesnt appear to be
supported.  XFS?? GFS??
2) Is LVM safe for this type of deployment?
3) Any other thoughts?

Really, does it have to be a single filesystem? Can you export a hierarchy of mounted smaller filesystems?

What is the largest filesystem you have now, and how long does it take to fsck? (I've not tried this either, but I'd not try it on an important system).

I think John has really raised an excellent point. I'm in a similar position, where users are requesting "huge" (by my standards) filesystems, and I find myself having to explain to them why they aren't a good idea, but that is based on my admittedly limited knowledge and experience.

I only want to use filesystems that are supported by Red Hat. (This shows that my experience is from the AIX and Solaris world. :-)

When we first started using RHEL, we wound up creating some 4, 6, and 8 TB filesystems (ext3, over LVM, from an FC SAN with SATABeast storage devices.) Due to our lack of experience in this environment, we wound up doing some things "wrong", and started experiencing semi- regular (once every 3-5 weeks) corruption on those filesystems. First we tried fsck'ing, and after waiting many hours for it to finish, only to find that it didn't fix all the problems, we wound up restoring from backup, which took days in some cases. Not exactly the experience I want to provide to my users.

Aside from that, there is the issue of backup. We have a well established and fairly robust backup system based on Tivoli Storage Manager. It works great. It also does its backups in parallel, but only on the filesystem level. When you've got an 8 TB filesystem, with 2-3 TB of changes daily, do you know what that does to your daily backups? I do: it hoses them, and give you heartburn.

it is certainly possible that I just don't know what I'm doing, but I don't think so. What I do think is that, while disk capacity has grown exponentially in the past few years, making multi-terabyte filesystems technically possible, in reality, all the tools the sysadmin needs to keep those filesystems healthy, and to fix them when their broken, just haven't kept pace.

(Not to hijack this subject, but I wish there was some resource or community that had experience in this area. I'm liberal in politics, but uber-conservative at work. I don't want to be the "first" to do something weird, like 8 TB filesystems. I want to learn from the pain and experience of others. :-)

So, to repeat John's point: do you really need to do it that way? Do you understand the issues you're going to face with backup and recovery? Do you users, or whomever is asking you do to it this way understand them?


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]