[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: optimising filesystem for many small files





On Sun, Oct 18, 2009 at 6:08 AM, Viji V Nair <viji fedoraproject org> wrote:
From: Matija Nalis <mnalis-ml voyager hr>

On Sun, Oct 18, 2009 at 03:01:46PM +0530, Viji V Nair wrote:
> The application which we are using are modified versions of mapnik and
> tilecache, these are single threaded so we are running 4 process at a

How does it scale if you reduce the number or processes - especially if you
run just one of those ? As this is just a single disk, 4 simultaneous
readers/writers would probably *totally* kill it with seeks.

I suspect it might even run faster with just 1 process then with 4 of
them...

with one process it is giving me 6 seconds

If it takes 6 seconds with one process and 20 seconds with 4 processes, then this
pretty clearly points to problems with thrashing the heads.
(this presumes that the timing you're mentioning is time between  request and
service with the same request patterns).

Others have suggested flash drives... This sounds like an idea.  On the cheaper end, I'd suggest  lots of mirroring.. The more drives the merrier. a 4-way mirrir will probably give you a good deal of speedup.   If you can find an old RAID box, try throwing in a dozen or so smaller drives (even 72 or 36GB SCSI drives)..

It sounds like the problem is clearly head seek, not transfer speeds, so lots of old SCSI drives with a single (slower) connection will probably do you more good than 4 demon-fast SATA drives. 
If you have smaller drives, go to raid10. If you have larger drives, then go to raid 1 and mirror them up the wazoo.  (I'm presuming that this is a read-mostly application.  If you're doing lots of parallel writes, then raid 10 might still be a good idea, even with big drives).

If you already have a deep mirror and you later get  Flash drives, then I'd say add the flash drives into the mix, rather than just replacing the RAID with flash, again -- unless this isn't a read-mostly situation -- the more drives the merrier.

Stephen Samuel http://www.bcgreen.com  Software, like love,
778-861-7641                              grows when you give it away

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]