[Pulp-list] Pulp RHEL Repo Download Forbidden??

Tue May 5 17:22:07 UTC 2015

I have seen this issue for months, a sync fails 9 times out of 10. It appears to be an issue (for me at least) on the Redhat side of things. I use curl to get more info.

I’ll run it one time and get a failure:

# curl -v —key ./Workstation-Entitlement.pem --cert ./Workstation-Entitlement.pem -k https://cdn.redhat.com/content/dist/rhel/workstation/7/7Workstation/x86_64/kickstart/treeinfo

  *
About to connect() to cdn.redhat.com port 443 (#0)
*   Trying 184.84.192.251...
* Connected to cdn.redhat.com (184.84.192.251) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* NSS: client certificate from file
* subject: CN=8a85f9894bd9c252014be203f1a6096f
* start date: Aug 01 04:00:00 2014 GMT
* expire date: Aug 01 03:59:59 2015 GMT
* common name: 8a85f9894bd9c252014be203f1a6096f
* issuer: E=ca-support at redhat.com,CN=Red Hat Candlepin Authority,OU=Red Hat Network,O="Red Hat, Inc.",ST=North Carolina,C=US
* SSL connection using TLS_RSA_WITH_AES_256_CBC_SHA
* Server certificate:
* subject: CN=cdn.redhat.com,OU=Red Hat Network,O=Red Hat,L=Raleigh,ST=North Carolina,C=US
* start date: May 14 19:48:02 2014 GMT
* expire date: May 11 19:48:02 2024 GMT
* common name: cdn.redhat.com
* issuer: E=ca-support at redhat.com,CN=Red Hat Entitlement Operations Authority,OU=Red Hat Network,O="Red Hat, Inc.",ST=North Carolina,C=US
> GET /content/dist/rhel/workstation/7/7Workstation/x86_64/kickstart/treeinfo HTTP/1.1
> User-Agent: curl/7.29.0
> Host: cdn.redhat.com
> Accept: */*
>
< HTTP/1.1 403 Forbidden
< Server: AkamaiGHost
< Mime-Version: 1.0
< Content-Type: text/html
< Content-Length: 369
< Expires: Tue, 05 May 2015 17:13:05 GMT
< Date: Tue, 05 May 2015 17:13:05 GMT
< X-Cache: TCP_DENIED from a128-241-218-165.deploy.akamaitechnologies.com (AkamaiGHost/7.2.0-15182023) (-)
< Connection: keep-alive
< EJ-HOST: edgejavaapp2.prod.a4.vary.redhat.com
< X-Akamai-Request-ID: 4a217f0
<
<HTML><HEAD>
<TITLE>Access Denied</TITLE>
</HEAD><BODY>
<H1>Access Denied</H1>

You don't have permission to access "http://cdn.redhat.com/content/dist/rhel/workstation/7/7Workstation/x86_64/kickstart/treeinfo" on this server.<P>
Reference #18.a5daf180.1430845985.4a217f0

And then I’ll re-run the command seconds later with a successful response:

# curl -v --key ./Workstation-Entitlement.pem --cert ./Workstation-Entitlement.pem -k https://cdn.redhat.com/content/dist/rhel/workstation/7/7Workstation/x86_64/kickstart/treeinfo
* About to connect() to cdn.redhat.com port 443 (#0)
*   Trying 184.84.192.251...
* Connected to cdn.redhat.com (184.84.192.251) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* NSS: client certificate from file
* subject: CN=8a85f9894bd9c252014be203f1a6096f
* start date: Aug 01 04:00:00 2014 GMT
* expire date: Aug 01 03:59:59 2015 GMT
* common name: 8a85f9894bd9c252014be203f1a6096f
* issuer: E=ca-support at redhat.com,CN=Red Hat Candlepin Authority,OU=Red Hat Network,O="Red Hat, Inc.",ST=North Carolina,C=US
* SSL connection using TLS_RSA_WITH_AES_256_CBC_SHA
* Server certificate:
* subject: CN=cdn.redhat.com,OU=Red Hat Network,O=Red Hat,L=Raleigh,ST=North Carolina,C=US
* start date: May 14 19:48:02 2014 GMT
* expire date: May 11 19:48:02 2024 GMT
* common name: cdn.redhat.com
* issuer: E=ca-support at redhat.com,CN=Red Hat Entitlement Operations Authority,OU=Red Hat Network,O="Red Hat, Inc.",ST=North Carolina,C=US
> GET /content/dist/rhel/workstation/7/7Workstation/x86_64/kickstart/treeinfo HTTP/1.1
> User-Agent: curl/7.29.0
> Host: cdn.redhat.com
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: Apache
< ETag: "11f6fa6eaa857d424b630447ab5334de:1424446169"
< Last-Modified: Fri, 20 Feb 2015 08:29:44 GMT
< Accept-Ranges: bytes
< Content-Length: 1471
< Content-Type: text/plain
< Date: Tue, 05 May 2015 17:16:10 GMT
< X-Cache: TCP_HIT from a128-241-218-165.deploy.akamaitechnologies.com (AkamaiGHost/7.2.0-15182023) (-)
< Connection: keep-alive
< EJ-HOST: rhej03.web.prod.ext.phx2.redhat.com
< X-Akamai-Request-ID: 4a57fb3
<
[checksums]
LiveOS/squashfs.img = sha256:198ef91d868e76c994680645964ef3873ec66fddb84be450370b051facaec8aa
images/pxeboot/initrd.img = sha256:101b3b5630b7032557be95aa8dcef50b01d8bfcdfa33429cea30fe09eaae9426
images/pxeboot/upgrade.img = sha256:03453b1f504e548ab9a933daa2f1fd440e48638f5deb9fac50be7dad929c1907
images/pxeboot/vmlinuz = sha256:67421a4877919ff0c16c27a53cba229e5f0771ae9cd32f3918caae2124a5a710
repodata/repomd.xml = sha256:014184dc5e503979a5577a97423e4340e5f71ac2746250bbdce91e0301b8c93f

…

I never have this issue syncing the Server repositories, only Workstation (and RHEL5 Client).

Reece

From: "Baird, Josh"
Date: Tuesday, May 5, 2015 at 4:23 AM
To: Gavin Jones, "pulp-list at redhat.com<mailto:pulp-list at redhat.com>"
Subject: Re: [Pulp-list] Pulp RHEL Repo Download Forbidden??

Hi Gavin,

I am having the same problem.  I just noticed that it was occurring yesterday.  I re-issued new entitlement certificates with valid expiration dates from RHN and the problem is still occurring.  I have verified that my certificates contain path/entitlements for the channels that I am trying to sync (via rct cat-cert).  Occasionally, Pulp will be able to download the metadata for certain channels, but then get 'Forbidden' when downloading individual packages.  Other times, it will throw a 'Forbidden' before being able to download the metadata as you pasted below.

I am going to hopefully spend some time working with the developers in #pulp today to get this figured out.  I have a feeling it is CDN related, but I'm not exactly sure at this point.

Thanks,

Josh

From: pulp-list-bounces at redhat.com<mailto:pulp-list-bounces at redhat.com> [mailto:pulp-list-bounces at redhat.com] On Behalf Of Gavin Jones
Sent: Tuesday, May 05, 2015 12:13 AM
To: pulp-list at redhat.com<mailto:pulp-list at redhat.com>
Subject: [Pulp-list] Pulp RHEL Repo Download Forbidden??

Hi Everyone, I seem to be getting an error when downloading from the Redhat Repos. This has only just stopped working and has been working fine for months.

It looks to be certificate related I believe from the logs.

* Firstly I have not changed anything on the pulp side
* I have checked my subscriptions are still active and the hosts that are connected to RHEL are still connected.

- Pulp Version:

rpm -qa | grep -i pulp

python-pulp-client-lib-2.6.0-1.el7.noarch
pulp-rpm-plugins-2.6.0-1.el7.noarch
python-pulp-bindings-2.6.0-1.el7.noarch
python-kombu-3.0.24-5.pulp.el7.noarch
python-isodate-0.5.0-4.pulp.el7.noarch
pulp-admin-client-2.6.0-1.el7.noarch
pulp-rpm-admin-extensions-2.6.0-1.el7.noarch
python-pulp-common-2.6.0-1.el7.noarch
pulp-server-2.6.0-1.el7.noarch
pulp-selinux-2.6.0-1.el7.noarch
python-pulp-rpm-common-2.6.0-1.el7.noarch

- Attempting to download the repo.

Please see below:

pulp-admin rpm repo sync run --repo-id=rhel-7-server-rhn-tools-rpms
+----------------------------------------------------------------------+
        Synchronizing Repository [rhel-7-server-rhn-tools-rpms]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[\]
... failed

Forbidden

Task Failed

Importer indicated a failed response

- Error Log

journalctl -f

ay 05 13:33:05 pulp01.rap.local pulp[2741]: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading metadata from https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/rhn-tools/os/.
May 05 13:33:05 pulp01.rap.local pulp[2741]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): cdn.redhat.com<http://cdn.redhat.com>
May 05 13:33:06 pulp01.rap.local pulp[2741]: pulp_rpm.plugins.importers.yum.sync:ERROR: (2741-28000) sync failed
May 05 13:33:06 pulp01.rap.local pulp[2741]: pulp_rpm.plugins.importers.yum.sync:ERROR: (2741-28000) Traceback (most recent call last):
May 05 13:33:06 pulp01.rap.local pulp[2741]: pulp_rpm.plugins.importers.yum.sync:ERROR: (2741-28000)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py",...e 104, in run
May 05 13:33:06 pulp01.rap.local pulp[2741]: pulp_rpm.plugins.importers.yum.sync:ERROR: (2741-28000)     metadata_files = self.get_metadata()
May 05 13:33:06 pulp01.rap.local pulp[2741]: pulp_rpm.plugins.importers.yum.sync:ERROR: (2741-28000)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py",... get_metadata
May 05 13:33:06 pulp01.rap.local pulp[2741]: pulp_rpm.plugins.importers.yum.sync:ERROR: (2741-28000)     raise FailedException(str(e))
May 05 13:33:06 pulp01.rap.local pulp[2741]: pulp_rpm.plugins.importers.yum.sync:ERROR: (2741-28000) FailedException: Forbidden
May 05 13:33:06 pulp01.rap.local pulp[2554]: celery.worker.job:ERROR: (2554-28000) Task pulp.server.managers.repo.sync.sync[81644b21-6bec-47dd-a31b-552baa2a27a8] raised unexpected: P...d response',)
May 05 13:33:06 pulp01.rap.local pulp[2554]: celery.worker.job:ERROR: (2554-28000) Traceback (most recent call last):
May 05 13:33:06 pulp01.rap.local pulp[2554]: celery.worker.job:ERROR: (2554-28000)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
May 05 13:33:06 pulp01.rap.local pulp[2554]: celery.worker.job:ERROR: (2554-28000)     R = retval = fun(*args, **kwargs)
May 05 13:33:06 pulp01.rap.local pulp[2554]: celery.worker.job:ERROR: (2554-28000)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 328, in __call__
May 05 13:33:06 pulp01.rap.local pulp[2554]: celery.worker.job:ERROR: (2554-28000)     return super(Task, self).__call__(*args, **kwargs)
May 05 13:33:06 pulp01.rap.local pulp[2554]: celery.worker.job:ERROR: (2554-28000)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
May 05 13:33:06 pulp01.rap.local pulp[2554]: celery.worker.job:ERROR: (2554-28000)     return self.run(*args, **kwargs)
May 05 13:33:06 pulp01.rap.local pulp[2554]: celery.worker.job:ERROR: (2554-28000)   File "/usr/lib/python2.7/site-packages/pulp/server/managers/repo/sync.py", line 114, in sync
May 05 13:33:06 pulp01.rap.local pulp[2554]: celery.worker.job:ERROR: (2554-28000)     raise PulpExecutionException(_('Importer indicated a failed response'))
May 05 13:33:06 pulp01.rap.local pulp[2554]: celery.worker.job:ERROR: (2554-28000) PulpExecutionException: Importer indicated a failed response
May 05 13:33:06 pulp01.rap.local pulp[2554]: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[e8f32211-ccc5-4918-b4d5-ada23e15ecf4] succeeded in 0.010533269s: None

is there a clean way to fix this issue without Deleting the entire repo and going through the process of setting this up again?

Thanks for your time.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20150505/bdbbdf82/attachment.htm>