[Pulp-list] Grinder orphanes packages after download
Chris St. Pierre
chris.a.st.pierre at gmail.com
Mon Nov 21 19:41:29 UTC 2011
Willem--
I ran into this issue, too. The long and the short of it is that the
HP packages are screwed up; for instance, the cpqacuxe package has a
filename that indicates that it's noarch, but in fact it's a i386
package, so grinder does the wrong thing (or maybe the too-right
thing) when trying to build a list of packages that are orphaned.
There are three options:
1. Get HP to fix their packages. And a pony, and the moon on a stick.
2. Grinder has an option that turns off orphaned package cleanup, but
it's not currently exposed to the Pulp layer. That could be done.
3. A far more complex approach that deals with inconsistent/broken
metadata and packages in a blindingly clever way.
I believe there's a ticket open and Pradeep is looking at something
along the lines of option 3. We currently only have one HP box
looking at Pulp, so this isn't a huge problem for us; if it becomes
more of an issue before there's a proper fix, I'll probably contribute
a patch to do option 2. Option 1 is obviously the best solution in
the same way that riding a unicorn to work is the best solution for
commuting.
TL;DR: There is no workaround currently, but the Pulp devs are aware
of the issue.
On Mon, Nov 21, 2011 at 1:16 PM, Willem Bos <bos.h.willem at gmail.com> wrote:
> Hi all,
>
> I'm trying to sync a HP hosted repository (downloads.linux.hp.com).
> Most of the packages are processed fine but a few are constantly
> orphaned by Grinder after being downloaded. A few of them do show up
> in /var/lib/pulp/packages/ but are not linked to a repo… I tried
> creating the repo with '--feed=http://...' and also with an added
> '--preserve_metada'.
>
> Then I downloaded the packages with reposync, created metadata with
> createrepo and placed this directory on a internal web server. With
> the same result as mentioned above.
>
> Finally, I tried '--feed=file:///...' (essentially bypassing Grinder)
> and all packages were assigned correctly.
>
> See below for detailed actions.
>
> Any suggestions?
>
> Regards,
> Willem.
>
> ## FIRST ATTEMPT: ##
>
> # pulp-admin repo create \
> --id=hp-test \
> --preserve_metadata \
> --feed=http://downloads.linux.hp.com/SDR/downloads/proliantsupportpack/redhat/6/packages/x86_64/
> Successfully created repository [ hp-test ]
>
> # pulp-admin repo sync --id=hp-test --threads=1 --foreground
> Sync for repository hp-test started
> Sync: Finished
> 46/46 new items downloaded
> 0/46 existing items processed
> Item Details:
> RPMs: 46/46
>
> # pulp-admin package search -f "^hp-.*"
> +------------------------------------------+
> Package Information
> +------------------------------------------+
> Name EVRA Filename
> Repositories
> hp-tg3 0:3.116j-3.src hp-tg3-3.116j-3.src.rpm
> hp-test
> hp-smh-templates 0:8.7.0.10-10.noarch
> hp-smh-templates-8.7.0.10-10.noarch.rpm hp-test
> hp-smh-templates 0:8.6.0.14-16.noarch
> hp-smh-templates-8.6.0.14-16.noarch.rpm hp-test
> ...
>
> # pulp-admin package search -f "^hp-snmp.*"
> error: No packages found.
>
> # ls -1d /var/lib/pulp/packages/hp-snmp*
> /var/lib/pulp/packages/hp-snmp-agents
>
> # cat /var/log/pulp/grinder.log
> ...
> 2011-11-20 21:13:35,155 13269:139726048700160: grinder.YumInfo:INFO:
> activeobject:160 46 packages have been marked to be fetched
> ...
> 2011-11-20 21:13:37,708 13269:139726048700160:
> grinder.ParallelFetch:INFO: ParallelFetch:177 46 items are marked to
> be fetched
> ...
> 2011-11-20 21:14:53,340 13269:139725956445952: grinder.BaseFetch:INFO:
> activeobject:160 Symlink missing in repo directory. Creating link
> /var/lib/pulp//repos/SDR/downloads/proliantsupportpack/redhat/6/packages/x86_64//hp-snmp-agents-8.6.2.3-9.x86_64.rpm
> to ../../../../../../../../packages/hp-snmp-agents/8.6.2.3/9/x86_64/e96/hp-snmp-agents-8.6.2.3-9.x86_64.rpm
> ...
> 2011-11-20 21:16:07,138 13269:139726048700160:
> grinder.ParallelFetch:INFO: ParallelFetch:238 ParallelFetch: 46 items
> successfully processed, 33 downloaded, 0 items had errors
> ...
> 2011-11-20 21:16:07,140 13269:139726048700160: grinder.RepoFetch:INFO:
> RepoFetch:154 Cleaning any orphaned packages..
> ...
> 2011-11-20 21:16:07,143 13269:139726048700160: grinder.RepoFetch:INFO:
> RepoFetch:190 Removing orphan package:
> /var/lib/pulp//repos/SDR/downloads/proliantsupportpack/redhat/6/packages/x86_64/hp-snmp-agents-8.6.2.3-9.x86_64.rpm
> ...
> 2011-11-20 21:16:07,144 13269:139726048700160: grinder.RepoFetch:INFO:
> RepoFetch:159 Processed <46> items in [154] seconds. Report: 46
> successes, 33 downloads, 0 errors
> 2011-11-20 21:16:
> ...
>
> # pulp-admin repo delete --id=hp-test
> Successful deleted repository [ hp-test ]
>
> # rm -r /var/lib/pulp/packages/{cpqacuxe,hpacucli,hpvca,hpdiags,hp-health,hp-snmp-agents}
>
> ## SECOND ATTEMPT: ##
>
> # cat > /etc/yum.repos.d/hp-test.repo << EOF
> [hp-test]
> name=hp-test
> baseurl=http://downloads.linux.hp.com/SDR/downloads/proliantsupportpack/redhat/6/packages/i686/
> enabled=1
> gpgcheck=0
> EOF
>
> # cd /var/www/html
>
> # reposync --repoid=hp-test .
> ...
> [hp-test: 11 of 31 ] Downloading hp-smh-templates-8.6.0.14-16.noarch.rpm
> hp-smh-templates-8.6.0.14-16.noarch.rpm | 3.4
> MB 00:02
> [hp-test: 12 of 31 ] Downloading hp-snmp-agents-8.6.2.3-8.rhel6.i386.rpm
> hp-snmp-agents-8.6.2.3-8.rhel6.i386.rpm | 1.7
> MB 00:00
> [hp-test: 13 of 31 ] Downloading hp-snmp-agents-8.7.0.23-16.rhel6.i386.rpm
> hp-snmp-agents-8.7.0.23-16.rhel6.i386.rpm | 1.7
> MB 00:00
> ...
>
> # cd hp-test
>
> # createrepo .
> Spawning worker 0 with 31 pkgs
> ...
>
> # zcat repodata/*-primary.xml.gz | grep "<name>hp-snmp"
> <name>hp-snmp-agents</name>
> <name>hp-snmp-agents</name>
>
> # pulp-admin repo create \
> --id=hp-test \
> --feed=http://n004.localdomain/hp-test/
>
> # pulp-admin repo sync --id=hp-test --threads=1 --foreground
>
> # pulp-admin package search -f "^hp-snmp.*"
> <output identical to first attempt>
>
> # cat /var/log/pulp/grinder.log
> <output identical to first attempt>
>
> # ls -1d /var/lib/pulp/packages/hp-*
> <output identical to first attempt>
>
> # pulp-admin repo delete --id=hp-test
> Successful deleted repository [ hp-test ]
>
> # rm -r /var/lib/pulp/packages/{cpqacuxe,hpacucli,hpdiags,hp-health,hp-snmp-agents}
>
> ## THIRD ATTEMPT: ##
>
> # pulp-admin repo create \
> --id=hp-test \
> --feed=file:///var/www/html/hp-test/
>
> # pulp-admin repo sync --id=hp-test --threads=1 --foreground
> pulp-admin repo sync --id=hp-test --threads=1 --foreground
> Sync for repository hp-test started
> Sync: Finished
> 31/31 new items downloaded
> 0/31 existing items processed
> Item Details:
> RPMs: 31/31
>
> # pulp-admin package search -f "^hp-.*"
> +------------------------------------------+
> Package Information
> +------------------------------------------+
> Name EVRA Filename
> Repositories
> hp-snmp-agents 0:8.7.0.23-16.i386
> hp-snmp-agents-8.7.0.23-16.rhel6.i386.rpm hp-test
> hp-snmp-agents 0:8.6.2.3-8.i386
> hp-snmp-agents-8.6.2.3-8.rhel6.i386.rpm hp-test
> ...
>
> _______________________________________________
> Pulp-list mailing list
> Pulp-list at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
>
--
Chris St. Pierre
More information about the Pulp-list
mailing list