[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Disk Space Issues - cvs-int/build hosts - (includes general note on Nagios Disk notifications)



Hey guys,

I seem to have deleted the nagios notification that I want to mention in particular, but as I noted in SOP/Nagios the %ages noted in the e-mails are what is left, so inode=99% doesn't mean that 99% of the inodes are used, it means 99% are still free.

Anyway, what this means, is that when nagios has been complaining about cvs-int recently, in particular the fact that /git has reached WARNING. After a bit of hunting around, I found /repo/pkgs using 168GiB of the 192GiB available, which is understandable, Fedora has got huge.

Problem here, is that there are a LOT of old tarballs in that folder, which leaves me wondering if we should do a spring clean ~1 mo after release.

Lets take banshee for example, a package which I adopted....

$ ls -l /repo/pkgs/banshee/
total 88
drwxrwsr-x  3 apache repoextras 4096 May  3  2006 banshee-0.10.10.tar.gz
drwxrwsr-x  3 apache repoextras 4096 Aug  7  2006 banshee-0.10.11.tar.gz
drwxrwsr-x  3 apache repoextras 4096 Aug 24  2006 banshee-0.10.12.tar.gz
drwxrwsr-x  3 apache repoextras 4096 Mar  4  2006 banshee-0.10.6.tar.gz
drwxrwsr-x  3 apache repoextras 4096 Mar  7  2006 banshee-0.10.7.tar.gz
drwxrwsr-x  3 apache repoextras 4096 Mar 14  2006 banshee-0.10.8.tar.gz
drwxrwsr-x  3 apache repoextras 4096 Apr 16  2006 banshee-0.10.9.tar.gz
drwxrwsr-x  3 apache repoextras 4096 Feb  2  2007 banshee-0.11.5.tar.bz2
drwxrwsr-x  3 apache repoextras 4096 Mar  7  2007 banshee-0.12.0.tar.bz2
drwxrwsr-x  3 apache repoextras 4096 Apr  5  2007 banshee-0.12.1.tar.bz2
drwxrwsr-x  3 apache repoextras 4096 Aug  7  2007 banshee-0.13.0.tar.bz2
drwxrwsr-x  3 apache repoextras 4096 Aug 31  2007 banshee-0.13.1.tar.bz2
drwxrwsr-x  3 apache repoextras 4096 Jan 14 15:58 banshee-0.13.2.tar.bz2
drwxrwsr-x  3 apache repoextras 4096 Apr 13 01:51 banshee-1-0.98.3.tar.bz2
drwxrwsr-x  3 apache repoextras 4096 May 10 03:41 banshee-1-0.99.1.tar.bz2

$ du -sh /repo/pkgs/banshee/
26M     /repo/pkgs/banshee/
(Approximately 2M/package)
At the most there should only be 4 tar balls there (R-2, R-1, R, Rawhide), R-2 only valid for one month after R has released.

Another couple of examples:
$ du -sh /repo/pkgs/kernel/
2.6G    /repo/pkgs/kernel/
(900ish tarballs dating back to 2004)

$ du -sh /repo/pkgs/kdeedu
1.4G    /repo/pkgs/kdeedu
(48 tarballs from KDE-3.0)

With plans such and Hans' plans of creating a 500M vegastrike data SRPM and the size growth and update schedules of some packages we are going to have these nightmares more and more frequently.

Two solutions I can think of:

Create a script, go thru ALL non dead.package's grab the tarball name from sources.list and basically create a bit of a database of what we are using, scan through /repo/pkgs and either:
* Move old tarballs to some archiving system (another use for archive.fp.o?)
* Delete old tarballs
Or throw more diskspace at cvs-int

Even if were to only remove 15% of the tarballs there (this is a very cautious estimate of the number of stale tarballs) we could potentially reach 72% diskspace available on that mount (down from 82%) - note this is very simplistic math, in essence, we could be no better off if we only removed the small stale tarballs :).

Diskspace isn't cheap, so I like delete old tarballs, I also like this option because it's not like they disappear completely, they should be in the src.rpm's already on archive.fp.o and if we accidentally delete 1 or 2 that are still needed, well grab it from src.rpm...

This leads on to my second item...

xenbuilder2 has run out of diskspace in /, it's down to 32M, thankfully koji has disabled it so it's safe for now, but wouldn't it be nice to throw say a 50GB partition dedicated solely for /var/lib/mock & /mnt/build? Yes, yes, I know money, but once again, builds are getting bigger so 'it'd be nice'.

Just thought I'd throw these two thoughts into the open, let the discussion begin!


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]