Better repodata performance

[David Farning]

seth vidal wrote:

> And how do you feed the repo creation system this data? Where do you get
> it to begin with? The only way you know this information is if you
> already have it - the only way you have it is if you checked all the
> packages for what has changed. Are you beginning to see the loop here?
> 

Hey, great work on yum!!

Let me see if I a understand the problem and can provide a general solution.

When running yum, a check is preformed to insure that the most recent 
versions of primary.xml is present, based on the timestamp found in 
repomd.xml.  If a more recent version of primary.xml is found then 
download it.

Now, in most situations very few packages have been changed. Therefore, 
between primary.xml versions few <package>...</packages> have changed 
thus most of the information in primary.xml is redundant against 
previous versions of primary.xml.

First, add a repo_build_number to repomd.xml.  This should should be an 
integer that increments every time createrepo is run.  This allows us to 
always know what build of a repo we are working with.

Createrepo:
saves the last runs primary.xml as primary.repo_build_number.xml.
	
Generates new primary.xml as usual.

Compares primary.xml to old primary.repo_build_number.xml to create 
primary.increment-(age).xml.  Where age indicates the packages changes 
required to bring a primary of age or less builds old up to date.

A number of primary.increment-(age).xml files could be made available.
Ages of 1, 5, 10 could be made available. These files could have a 
'rolling' age to cover a majority of updates without wasting too much 
space.

On the Yum side of things:
Read local repomd.xml to determine local repo_build_number
dl new repomd.xml to determine current repo_build_number.

if remote repo_build_number = local repo_build_number = 0
	local primary.xml is uptodate

if remote repo_build_number = local repo_build_number = 1
	dl primary.increment-1.xml
	merge primary.increment-1.xml with local primary.xml

if remote repo_build_number = local repo_build_number <= 5
	dl primary.increment-5.xml
	merge primary.increment-5.xml with local primary.xml

if remote repo_build_number = local repo_build_number <= 10
	dl primary.increment-10.xml
	merge primary.increment-1.xml with local primary.xml

if remote repo_build_number = local repo_build_number > 10
	dl primary.xml


-dtf