Fedora Updates System

[Luke Macken]

On Fri, Nov 24, 2006 at 11:09:58AM +0100, Michael Schwendt wrote:
[...]
> repoview
> --------
> 
> Running it for all of Fedora Extras takes a lot of time. So long, that
> even if you let the push script do its work in a background terminal, it
> is still painful to see how long it takes to complete.
> 
> For unknown reasons we also create repoview pages for the "debug"
> repositories. If it were just my own decision, I would stop doing that,
> since I doubt those web pages are popular enough. Who really browses
> repoview for debuginfo packages? We should expect debuginfo packages to be
> available for every relevant package in the repository.

Yeah, repoview for debuginfo packages seems unnecessary.  We don't even run
repoview for updates{,-testing} at the moment, but we can easily integrate
it with the new system if we want.

> createrepo < 0.4.5 is unable to handle "unknown" files in its repodata
> directory. Therefore it conflicts with repoview and runs into a fatal
> error condition with a premature program termination, leaving behind a
> temporary ".olddir". This is especially ugly, since an administrator would
> need to recover from that manually and either move back files from
> ".olddir" or delete it. But when deleted, the repoview tree is lost and is
> created from scratch. I've been told that this is a problem for mirrors,
> where the several thousands of files are re-examined for changes just
> because of the fresh time stamps. So, as of a few weeks ago, the push
> script works around that successfully with a repodata backup strategy
> outside of createrepo.

The old updates system code (and the fedora-updates-clean  cronjob) has hacks
around this, as well, when dealing with the updateinfo.xml.gz.  I
haven't checked, but how does createrepo >= 0.4.5 deal with unknown
files now?

> For createrepo and repoview to be run in the background as a scheduled
> job, it needs a local lock on the repository. Most likely _not_
> fine-grained locking on every arch-specific sub-repository, because every
> locking comes at a cost (especially when there are multiple jobs of
> different priority waiting).

Yeah, there definitely needs to be mutual exclusion with the updates
staging repository.

> repoclosure
> -----------
> 
> Running this takes even more time. Currently, it examine all of Fedora
> Extras + Core + Updates + Legacy in a background job after packages have
> been published. The time it takes is approximately the difference between
> the time stamps of the build report and the broken deps report.

I'd like to run repoclosure on the updates before pushing them out at
all (sure we have updates-testing, but I don't want that becoming
another rawhide).

I spoke with skvidal over the summer speeding up the
closure process, and seem to remember mention of caching the
provides/requires to gain some speed.  We'll definitely have to look
into this, as this process is indeed dreadfully slow.

> > ''' Pushing '''
> >  * Moves packages to proper updates stage
> 
> More "stages" which are understood by plague would be good. We only use
> one stage, needsign, which is the build-results repository known to the
> build servers.

Well, once we get the Brew situation figured out, we can use its
detached gpg signatures to help with the push process.  Whatever stages
we want (pending/testing/needsign/pushed) should probably be implemented
in the updates system itself, to allow the buildsystem to only worry about
spitting out builds.  Althought, my Brew-fu is weak, so we'll have to
see what it offers once/if it is released.

[..]
> >  * updates repo cleaner
> >  * remove old packages
> 
> So far, the script I named repoprune is much faster than repomanage and
> simplifies Fedora Extras repository maintenance a lot, since it gets rid
> of orphaned and out-of-date sub-packages automatically, too.

Awesome, I will see what I can do about integrating this tool into the
new system.


luke