[Spacewalk-list] Need for a lightweight mirroring solution

[Michael DeHaan]

Mike McCune wrote:
> Michael DeHaan wrote:
>> Covil, K (Kim) wrote:
>>>> From: spacewalk-list-bounces at redhat.com [mailto:spacewalk-list-
>>>> bounces at redhat.com] On Behalf Of Michael DeHaan
>>>> Sent: 21 July 2008 13:38
>>>> To: Jon Stanley
>>>> Cc: spacewalk-list at redhat.com
>>>> Subject: Re: [Spacewalk-list] Need for a lightweight mirroring 
>>>> solution
>>>>
>>>> Jon Stanley wrote:
>>>>    
>>>>> On Mon, Jul 21, 2008 at 8:05 AM, Michael DeHaan <mdehaan at redhat.com>
>>>>>       
>>>> wrote:
>>>>    
>>>>>> Many questions I get on cobbler's mailing list deal with using
>>>>>> yum_rhn_plugin to mirror RHEL packages.   These are typically from
>>>>>>         
>>>> users who
>>>>    
>>>>>> have small-mid size environments that cannot use Satellite for
>>>>>>         
>>>> various
>>>>    
>>>>>> reasons.
>>>>>>
>>>>>>         
>>>>> That;s why we have spacewalk now :)
>>>>>
>>>>>
>>>>>       
>>>> Well, no, the problem is still the same.    The small disconnected 
>>>> labs
>>>> are looking to a solution, and they need to buy Satellite in order to
>>>> do
>>>> mirroring, even if they only have 10 systems.
>>>>
>>>>     
>>> This seems like a perfect situation for the suggestion made earlier 
>>> about allowing Spacewalk to refer to external repositories. Use 
>>> squid to only cache the packages required by the 10 systems rather 
>>> than downloading the entire RHEL/CentOS repository, and just handle 
>>> package lists in Spacewalk rather than the packages themselves.
>>>   
>>
>
> I like the above ideas as well.  I've wanted to add on to Pulp's CLI 
> and make a stand alone tool for content synchronization.  A while back 
> I threw up some updated info in the Pulp wiki with my thoughts on what 
> I thought we might need in a pulp CLI tool:
>
> https://fedorahosted.org/pulp
>
> ""PULP ¶
>
> Pulp is an an application for managing software content. Its goals are:
>
>     * The replication software repositories from any location (http, 
> file system, ISO, RHN) to a local onsite repository.
>     * Provide and easy mechanism for systems to get access to these 
> repositories and install software from them.
>     * An accounting mechanism that keeps track of what systems are 
> using what repositories
>
> At its core we envision pulp being able to provide a series of input 
> and output formats to synchronize content to and from various locations.
>
> Examples would be:
>
> Mirror F9 i386 everything repository:
>
>   pulp mirror 
> --input=http://download.fedora.redhat.com/pub/fedora/linux/releases/9/Everything/i386/os/Packages/\ 
>
>     --output=file:///var/www/repo/f9-i386
>
> Mirror Red Hat Enterprise Linux version 5 to an ISO:
>
> pulp mirror --input=rhn://rhel-i386-server-5 
> --output=iso://root/RHEL5-PACKAGES.ISO
>
> Mirror Fedora 9 into a Spacewalk Channel:
>
>   pulp mirror 
> --input=http://download.fedora.redhat.com/pub/fedora/linux/releases/9/Everything/i386/os/Packages/\ 
>
>     --output=spacewalk://somespacewalk.example.com  --username=admin 
> --password=redhat
>
> Pulp will provide an easy web, web-services, and command line 
> interface for managing all of this.""
>
> That said, my personal bandwidth for implementing such a project is 
> somewhat low, although as others have said, the desire for things like 
> above is definitely there.
>
> The central problem we still have is that the only way to synchronize 
> all the current, old and various arches of RHEL to a machine is by the 
> use of satellite-sync to an RHN Satellite.
>
> If we are looking to synchronize RHEL, we will need to use the 
> satellite-sync API or work with the Red Hat Network team to provide 
> new APIs for any new tools we develop.
>
> If we want a tool to work with things *other* than RHEL then we are in 
> better shape because the content is easier to get too.
>
> With Spacewalk users are currently tasked with fetching their own 
> content and using rhnpush to get the content into a Spacewalk channel. 
> While this is workable, it isn't the most user friendly of methods.   
> I think there still is a need for a tool for fetching remote content 
> and keeping it in sync with some form of local storage (disk or 
> spacewalk channel) which is what Dehaan and the Cobbler users are 
> clamoring for.
>
> I think we should develop a new tool like described above for content 
> retrieval and use Spacewalk channels to track usage.
>

++.

I would be interested in working on something like this too, but also 
have some Cobbler commitments.    Maybe we should steal some cycles from 
each and get this going as I don't think it would be too difficult 
seeing we know mostly what is involved.

> As for the feature to make Spacewalk Channels be proxies for remote 
> content, I think that is a good thing but doesn't help the users who 
> want all the content available locally.
>
>> That doesn't solve the disconnected lab case, but probably does solve 
>> the problem where you need to route through a system in a DMZ for 
>> updates -- probably 90% of the cases we need to solve.   I like this.
>
> Disconnected labs will need to sync all the content to some local 
> store for sure.  A CLI tool like proposed above really does help with 
> this case.
>
> The thing about using Spacewalk Proxy is that it still requires either 
> a Spacewalk server, a RHN Satellite or a connection to rhn.redhat.com 
> for it to function.  It has no notion of really what a channel is or 
> any local 'management' of the content.  It is a glorified squid cache.
>
>> If we solve it generically, this could also possibly allow using 
>> Spacewalk Proxy to be a Proxy for Red Hat Network as well as just a 
>> Proxy for another Spacewalk/Satellite?  
>
> Not quite following this statement ...

Basically this was about whether we could use the Proxy without the 
"Spacewalk Server", which we can't now, but might be interesting.

That being said, i don't think caching is the architectural answer we 
want -- I like to have real and fully complete repos, as that way you do 
not need to rely on the network connection outside being reliable [sic] 
or fast.


>
> Mike