Single Entry ReWrite Code - foo.rpm

[Matt Domsch]

On Thu, Feb 22, 2007 at 02:23:57PM -0700, Jonathan Steffan wrote:
>    Team,
> 
>        I've been working on getting jigdo usable for Fedora Unity. In the
>    process, I needed a way to have a single url to access rpms on public
>    mirrors.

Mirrormanager has code to do this right now if you want it.  (Granted,
it's not in use yet, but soon...).  The URLs underneath /pub/.... on
the mirrormanager URL provide the list of up-to-date mirrors that have
that content.  Right now it's all directory-based, but could easily be
extended to be file-based.  By that I mean:

http://admin.fedora.redhat.com/mirrormanager/pub/fedora/linux/core/6/i386/iso/

will return the list of mirrors containing the content of
pub/fedora/linux/core/6/i386/iso/
that's up-to-date.

> Assume the requested file is foo.rpm What I have done so far is
>    setup a rewrite map that does the following:
> 
>      * Pull the mirrorlist while preserving the requesting IP as
>        X-Fowarded-For (for geoIP)
>      * Parse the mirrorlist
>      * Random loop the mirrorlist and request the HEAD for foo.rpm from a
>        given mirror, if not 404 continue

Is the code for this piece available?  Does it use keepalives? :-)
The keepalives are what tried to hack into the mirrormanager crawler
earlie this week.  (urlgrabber's keepalive.py is close to what I need,
but I need it to do HEADs not GETs, so I was overriding various parts
and it got messy in a hurry).

>      * Redirect the request for foo.rpm to a public mirror (using 302) that
>        has verified it has the file
> 
>    Things I plan on doing:
> 
>      * Cache results (assuming no DB backend, do RAM caching based on
>        session; IP, remote mirror, etc)
>      * Maybe rate public mirrors based on:
> 
>           * Number of missing files (404)
>           * Latency (granted this is from the rewrite server)
>           * Bitrate test (ran every hour or something, also from the rewrite
>             server)
> 
>      * Setup a database that can be prepopulated with this data, potentially
>        using data from mirrormanager

:-)

> 
>    If needed, round robin would help keep things going.. or even just use
>    pound (or something else) between two machines.
> 
>    Any thoughts? I'd like to have Fedora Project to provide this feature. If
>    not, I will be setting it up anyways.
> 
>    Jonathan Steffan
>    daMaestro

if mirrormanager can provide what you need, I'm sure open to
contributions (ideas and/or patches).

Thanks,
Matt

-- 
Matt Domsch
Software Architect
Dell Linux Solutions linux.dell.com & www.dell.com/linux
Linux on Dell mailing lists @ http://lists.us.dell.com