[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: Fedora extras metadata



Matt Domsch wrote:

First, the problems with the current mirroring is really twofold:

( ... snip ... )
2) the global mirrors aren't being notified when content has
   changed on the master, such that they should start a new rsync
   run.  mirrormanager takes a per-host email address, which the
   master sign-and-push scripts will eventually send an email to when
   the content changes.  As it stands, syncing every 6 hours when
   nothing has changed doesn't make any sense.

Now, mirrormanager has 2 methods by which it can know a give mirror
host is up-to-date.  First is a new report_mirrors script that uploads
directory data back to the database from the mirror itself.  Not
everyone will want to run that, and there's always the 'trust but
verify' model, so I've also got a (fast?) crawler that crawls each
host using HTTP HEADs and keepalives or FTP DIR calls looking for
content it should be carrying compared to the master list, and
tracking presence and up-to-date-ness on a per-directory level.  Those
that aren't up2date get dropped from the appropriate per-directory
lists (e.g. the repodata dirs) in real time.


Sounds good, but it still seems like we need special tools to determine on the fly, with standard tools, if a mirror is in a sync'ed state. If I need to create a local mirror of fedora X with updates, with standard tools, I have to figure out when to mirror by trial and error and have no way to check before the sync, if the mirror is in the middle of syncing itself.

The mirrormanager you're working on will be useful for monitoring and flagging stale mirrors. It could work well for yum too, but for the basic mirroring, we still lack a simple mechanism to allow us to track the state of upstream mirrors. Mirroring a few gigs of inconsistent data makes no sense at all and this can be implemented very very easily. It's all a matter of documenting "HOWTO setup a fedora mirror".

That's the idea.  A lot of the code is implemented, there's more to
go.  If you're good with python, turbogears, and the like, I'm sure I
could put you to work on it.  Drop me a note.


Never worked with turbogears, sorry ;-)

/Thomas


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]