[Pulp-list] Grinder orphanes packages after download
Willem Bos
bos.h.willem at gmail.com
Mon Nov 21 20:33:52 UTC 2011
Hi Chris,
Thanks for your quick reply. I can't wait to see the solution for
option three, would love to see the .diff for option 2 and... well I
don't care much for ponies :-)
Regards,
Willem.
On Mon, Nov 21, 2011 at 8:41 PM, Chris St. Pierre
<chris.a.st.pierre at gmail.com> wrote:
> Willem--
>
> I ran into this issue, too. The long and the short of it is that the
> HP packages are screwed up; for instance, the cpqacuxe package has a
> filename that indicates that it's noarch, but in fact it's a i386
> package, so grinder does the wrong thing (or maybe the too-right
> thing) when trying to build a list of packages that are orphaned.
>
> There are three options:
>
> 1. Get HP to fix their packages. And a pony, and the moon on a stick.
>
> 2. Grinder has an option that turns off orphaned package cleanup, but
> it's not currently exposed to the Pulp layer. That could be done.
>
> 3. A far more complex approach that deals with inconsistent/broken
> metadata and packages in a blindingly clever way.
>
> I believe there's a ticket open and Pradeep is looking at something
> along the lines of option 3. We currently only have one HP box
> looking at Pulp, so this isn't a huge problem for us; if it becomes
> more of an issue before there's a proper fix, I'll probably contribute
> a patch to do option 2. Option 1 is obviously the best solution in
> the same way that riding a unicorn to work is the best solution for
> commuting.
>
> TL;DR: There is no workaround currently, but the Pulp devs are aware
> of the issue.
>
> On Mon, Nov 21, 2011 at 1:16 PM, Willem Bos <bos.h.willem at gmail.com> wrote:
>> Hi all,
>>
>> I'm trying to sync a HP hosted repository (downloads.linux.hp.com).
>> Most of the packages are processed fine but a few are constantly
>> orphaned by Grinder after being downloaded. A few of them do show up
>> in /var/lib/pulp/packages/ but are not linked to a repo… I tried
>> creating the repo with '--feed=http://...' and also with an added
>> '--preserve_metada'.
>>
>> Then I downloaded the packages with reposync, created metadata with
>> createrepo and placed this directory on a internal web server. With
>> the same result as mentioned above.
>>
>> Finally, I tried '--feed=file:///...' (essentially bypassing Grinder)
>> and all packages were assigned correctly.
>>
>> See below for detailed actions.
>>
>> Any suggestions?
>>
>> Regards,
>> Willem.
>>
>> ## FIRST ATTEMPT: ##
>>
>> # pulp-admin repo create \
>> --id=hp-test \
>> --preserve_metadata \
>> --feed=http://downloads.linux.hp.com/SDR/downloads/proliantsupportpack/redhat/6/packages/x86_64/
>> Successfully created repository [ hp-test ]
>>
>> # pulp-admin repo sync --id=hp-test --threads=1 --foreground
>> Sync for repository hp-test started
>> Sync: Finished
>> 46/46 new items downloaded
>> 0/46 existing items processed
>> Item Details:
>> RPMs: 46/46
>>
>> # pulp-admin package search -f "^hp-.*"
>> +------------------------------------------+
>> Package Information
>> +------------------------------------------+
>> Name EVRA Filename
>> Repositories
>> hp-tg3 0:3.116j-3.src hp-tg3-3.116j-3.src.rpm
>> hp-test
>> hp-smh-templates 0:8.7.0.10-10.noarch
>> hp-smh-templates-8.7.0.10-10.noarch.rpm hp-test
>> hp-smh-templates 0:8.6.0.14-16.noarch
>> hp-smh-templates-8.6.0.14-16.noarch.rpm hp-test
>> ...
>>
>> # pulp-admin package search -f "^hp-snmp.*"
>> error: No packages found.
>>
>> # ls -1d /var/lib/pulp/packages/hp-snmp*
>> /var/lib/pulp/packages/hp-snmp-agents
>>
>> # cat /var/log/pulp/grinder.log
>> ...
>> 2011-11-20 21:13:35,155 13269:139726048700160: grinder.YumInfo:INFO:
>> activeobject:160 46 packages have been marked to be fetched
>> ...
>> 2011-11-20 21:13:37,708 13269:139726048700160:
>> grinder.ParallelFetch:INFO: ParallelFetch:177 46 items are marked to
>> be fetched
>> ...
>> 2011-11-20 21:14:53,340 13269:139725956445952: grinder.BaseFetch:INFO:
>> activeobject:160 Symlink missing in repo directory. Creating link
>> /var/lib/pulp//repos/SDR/downloads/proliantsupportpack/redhat/6/packages/x86_64//hp-snmp-agents-8.6.2.3-9.x86_64.rpm
>> to ../../../../../../../../packages/hp-snmp-agents/8.6.2.3/9/x86_64/e96/hp-snmp-agents-8.6.2.3-9.x86_64.rpm
>> ...
>> 2011-11-20 21:16:07,138 13269:139726048700160:
>> grinder.ParallelFetch:INFO: ParallelFetch:238 ParallelFetch: 46 items
>> successfully processed, 33 downloaded, 0 items had errors
>> ...
>> 2011-11-20 21:16:07,140 13269:139726048700160: grinder.RepoFetch:INFO:
>> RepoFetch:154 Cleaning any orphaned packages..
>> ...
>> 2011-11-20 21:16:07,143 13269:139726048700160: grinder.RepoFetch:INFO:
>> RepoFetch:190 Removing orphan package:
>> /var/lib/pulp//repos/SDR/downloads/proliantsupportpack/redhat/6/packages/x86_64/hp-snmp-agents-8.6.2.3-9.x86_64.rpm
>> ...
>> 2011-11-20 21:16:07,144 13269:139726048700160: grinder.RepoFetch:INFO:
>> RepoFetch:159 Processed <46> items in [154] seconds. Report: 46
>> successes, 33 downloads, 0 errors
>> 2011-11-20 21:16:
>> ...
>>
>> # pulp-admin repo delete --id=hp-test
>> Successful deleted repository [ hp-test ]
>>
>> # rm -r /var/lib/pulp/packages/{cpqacuxe,hpacucli,hpvca,hpdiags,hp-health,hp-snmp-agents}
>>
>> ## SECOND ATTEMPT: ##
>>
>> # cat > /etc/yum.repos.d/hp-test.repo << EOF
>> [hp-test]
>> name=hp-test
>> baseurl=http://downloads.linux.hp.com/SDR/downloads/proliantsupportpack/redhat/6/packages/i686/
>> enabled=1
>> gpgcheck=0
>> EOF
>>
>> # cd /var/www/html
>>
>> # reposync --repoid=hp-test .
>> ...
>> [hp-test: 11 of 31 ] Downloading hp-smh-templates-8.6.0.14-16.noarch.rpm
>> hp-smh-templates-8.6.0.14-16.noarch.rpm | 3.4
>> MB 00:02
>> [hp-test: 12 of 31 ] Downloading hp-snmp-agents-8.6.2.3-8.rhel6.i386.rpm
>> hp-snmp-agents-8.6.2.3-8.rhel6.i386.rpm | 1.7
>> MB 00:00
>> [hp-test: 13 of 31 ] Downloading hp-snmp-agents-8.7.0.23-16.rhel6.i386.rpm
>> hp-snmp-agents-8.7.0.23-16.rhel6.i386.rpm | 1.7
>> MB 00:00
>> ...
>>
>> # cd hp-test
>>
>> # createrepo .
>> Spawning worker 0 with 31 pkgs
>> ...
>>
>> # zcat repodata/*-primary.xml.gz | grep "<name>hp-snmp"
>> <name>hp-snmp-agents</name>
>> <name>hp-snmp-agents</name>
>>
>> # pulp-admin repo create \
>> --id=hp-test \
>> --feed=http://n004.localdomain/hp-test/
>>
>> # pulp-admin repo sync --id=hp-test --threads=1 --foreground
>>
>> # pulp-admin package search -f "^hp-snmp.*"
>> <output identical to first attempt>
>>
>> # cat /var/log/pulp/grinder.log
>> <output identical to first attempt>
>>
>> # ls -1d /var/lib/pulp/packages/hp-*
>> <output identical to first attempt>
>>
>> # pulp-admin repo delete --id=hp-test
>> Successful deleted repository [ hp-test ]
>>
>> # rm -r /var/lib/pulp/packages/{cpqacuxe,hpacucli,hpdiags,hp-health,hp-snmp-agents}
>>
>> ## THIRD ATTEMPT: ##
>>
>> # pulp-admin repo create \
>> --id=hp-test \
>> --feed=file:///var/www/html/hp-test/
>>
>> # pulp-admin repo sync --id=hp-test --threads=1 --foreground
>> pulp-admin repo sync --id=hp-test --threads=1 --foreground
>> Sync for repository hp-test started
>> Sync: Finished
>> 31/31 new items downloaded
>> 0/31 existing items processed
>> Item Details:
>> RPMs: 31/31
>>
>> # pulp-admin package search -f "^hp-.*"
>> +------------------------------------------+
>> Package Information
>> +------------------------------------------+
>> Name EVRA Filename
>> Repositories
>> hp-snmp-agents 0:8.7.0.23-16.i386
>> hp-snmp-agents-8.7.0.23-16.rhel6.i386.rpm hp-test
>> hp-snmp-agents 0:8.6.2.3-8.i386
>> hp-snmp-agents-8.6.2.3-8.rhel6.i386.rpm hp-test
>> ...
>>
>> _______________________________________________
>> Pulp-list mailing list
>> Pulp-list at redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-list
>>
>
>
>
> --
> Chris St. Pierre
>
More information about the Pulp-list
mailing list