Better repodata performance

[Dag Wieers]

On Mon, 7 Feb 2005, Dag Wieers wrote:

> On Sun, 29 Jan 2005, Alexandre Oliva wrote:
> > On Jan 29, 2005, "Charles R. Anderson" <cra at WPI.EDU> wrote:
> > 
> > > Why all the complication when a general-purpose algorithm, rsync,
> > > already exists to solve this problem?
> > 
> > It doesn't do very well on compressed files, and there aren't a lot of
> > rsync-enabled web proxies out there.
> 
> You may like this RFE:
> 
> 	https://devel.linux.duke.edu/bugzilla/show_bug.cgi?id=391
> 
> gzip has an --rsyncable option that is missing from the python zlib 
> implementation. If python/zlib could be taught this, and createrepo 
> is able to use it, the metadata would be rsyncable (improves 
> transferspeed for repo maintainers/mirrors).

My repo-generation script now unzips the createrepo metadata, recompresses 
with --rsyncable and recreates the repomd.xml after createrepo's run.

I synced the repository.
I then added a single update package to my repository
And then rsynced again (with a 12kB upstream limitation), and got:

	fedora/3/en/i386/dag/repodata/
	fedora/3/en/i386/dag/repodata/filelists.xml.gz
	      961353 100%  306.10kB/s    0:00:03  (68, 16.8% of 208800)
	fedora/3/en/i386/dag/repodata/other.xml.gz
	      412335 100%   89.30kB/s    0:00:04  (69, 16.8% of 208800)
	fedora/3/en/i386/dag/repodata/primary.xml.gz
	      668753 100%  154.79kB/s    0:00:04  (70, 16.8% of 208800)
	fedora/3/en/i386/dag/repodata/repomd.xml
	        1128 100%    4.87kB/s    0:00:00  (71, 16.8% of 208800)
and
	fedora/3/en/x86_64/dag/repodata/
	fedora/3/en/x86_64/dag/repodata/filelists.xml.gz
	     1466742 100%  273.93kB/s    0:00:05  (86, 20.9% of 208800)
	fedora/3/en/x86_64/dag/repodata/other.xml.gz
	      674072 100%   79.47kB/s    0:00:08  (87, 20.9% of 208800)
	fedora/3/en/x86_64/dag/repodata/primary.xml.gz
	     1086239 100%  141.68kB/s    0:00:07  (88, 20.9% of 208800)
	fedora/3/en/x86_64/dag/repodata/repomd.xml
	        1128 100%    1.46kB/s    0:00:00  (89, 20.9% of 208800)

Which is a between 6.6x and 25.6x speed improvement for something I have 
to update every single sync. (Normally these files are synced with a 
12kB/sec rate limitation taking on average 1min per file, now only 6secs)

Now imagine what this would do if Yum had a client-side rsync 
implementation. zsync may be something to look at.

--   dag wieers,  dag at wieers.com,  http://dag.wieers.com/   --
[all I want is a warm bed and a kind word and unlimited power]