Fedora extras metadata

[Thomas M Steenholdt]

Matt Domsch wrote:
> 
> First, the problems with the current mirroring is really twofold:
> 
( ... snip ... )
> 2) the global mirrors aren't being notified when content has
>    changed on the master, such that they should start a new rsync
>    run.  mirrormanager takes a per-host email address, which the
>    master sign-and-push scripts will eventually send an email to when
>    the content changes.  As it stands, syncing every 6 hours when
>    nothing has changed doesn't make any sense.
> 
> Now, mirrormanager has 2 methods by which it can know a give mirror
> host is up-to-date.  First is a new report_mirrors script that uploads
> directory data back to the database from the mirror itself.  Not
> everyone will want to run that, and there's always the 'trust but
> verify' model, so I've also got a (fast?) crawler that crawls each
> host using HTTP HEADs and keepalives or FTP DIR calls looking for
> content it should be carrying compared to the master list, and
> tracking presence and up-to-date-ness on a per-directory level.  Those
> that aren't up2date get dropped from the appropriate per-directory
> lists (e.g. the repodata dirs) in real time.
> 

Sounds good, but it still seems like we need special tools to determine 
on the fly, with standard tools, if a mirror is in a sync'ed state. If I 
need to create a local mirror of fedora X with updates, with standard 
tools, I have to figure out when to mirror by trial and error and have 
no way to check before the sync, if the mirror is in the middle of 
syncing itself.

The mirrormanager you're working on will be useful for monitoring and 
flagging stale mirrors. It could work well for yum too, but for the 
basic mirroring, we still lack a simple mechanism to allow us to track 
the state of upstream mirrors. Mirroring a few gigs of inconsistent data 
makes no sense at all and this can be implemented very very easily. It's 
all a matter of documenting "HOWTO setup a fedora mirror".

> That's the idea.  A lot of the code is implemented, there's more to
> go.  If you're good with python, turbogears, and the like, I'm sure I
> could put you to work on it.  Drop me a note.
> 

Never worked with turbogears, sorry ;-)

/Thomas