[Pulp-list] pulp iso sync failure
ben.stanley at exemail.com.au
ben.stanley at exemail.com.au
Wed May 13 07:58:09 UTC 2015
Hello pulp-list,
I have just observed a repeatable pulp sync failure of an ISO repo, where
curl managed to fetch the PULP_MANIFEST successfully.
The pulp sync failure reason appears to be the following log message:
May 13 17:38:11 bumblebee pulp: pulp_rpm.plugins.importers.iso.sync:ERROR:
Failed to download
https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/iso/PULP_MANIFEST:
('Connection aborted.', error(110, 'Connection timed out')).
The timeout occurs 2 min after the previous log message.
The following list summarises all the gory details that can be found at
the end of this email.
* pulp-admin iso repo create
* pulp-admin iso repo sync
* pulp.log containing failure
* successful curl command
* pulp rpm versions
I am hoping that someone will be able to help me to sync the repo
successfully, or reproduce the problem and initiate a fix.
Thanks,
Ben Stanley.
=========== pulp-admin iso repo create ===================================
[root at bumblebee conf.d]# pulp-admin iso repo create \
--repo-id=rhel-7-server-x86_64-iso \
--feed=https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/iso
\
--feed-ca-cert=/etc/rhsm/ca/redhat-uep.pem \
--feed-cert=/etc/pki/entitlement/$keyid.pem \
--feed-key=/etc/pki/entitlement/$keyid-key.pem \
--validate true \
--serve-http true
Successfully created repository [rhel-7-server-x86_64-iso]
==========================================================================
=========== pulp-admin iso repo sync run =================================
[root at bumblebee conf.d]# date
Wed May 13 17:35:56 AEST 2015
[root at bumblebee conf.d]# pulp-admin iso repo sync run
--repo-id=rhel-7-server-x86_64-iso
+----------------------------------------------------------------------+
Synchronizing Repository [rhel-7-server-x86_64-iso]
+----------------------------------------------------------------------+
This command may be exited via ctrl+c without affecting the request.
Downloading the Pulp Manifest...
Downloading the Pulp Manifest failed:
{}
Task Failed
Importer indicated a failed response
[root at bumblebee conf.d]# date
Wed May 13 17:38:23 AEST 2015
[root at bumblebee conf.d]#
==========================================================================
=========== /var/log/pulp ================================================
May 13 17:36:00 bumblebee pulp: celery.worker.strategy:INFO: Received
task:
pulp.server.async.tasks._queue_reserved_task[0e90b376-d589-4261-bf8f-5afcb8514c91]
May 13 17:36:00 bumblebee pulp: celery.worker.strategy:INFO: Received
task:
pulp.server.managers.repo.sync.sync[0e0bbc76-43de-4dd4-b45c-edff18a59644]
May 13 17:36:00 bumblebee pulp: celery.worker.strategy:INFO: Received
task:
pulp.server.async.tasks._release_resource[09882136-6b6b-422d-bd55-3d312943cad3]
May 13 17:36:00 bumblebee pulp:
requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS
connection (1): cdn.redhat.com
May 13 17:36:00 bumblebee pulp: celery.worker.job:INFO: Task
pulp.server.async.tasks._queue_reserved_task[0e90b376-d589-4261-bf8f-5afcb8514c91]
succeeded in 0.0365472619887s: None
May 13 17:38:11 bumblebee pulp: nectar.downloaders.threaded:ERROR:
(17541-47008) ('Connection aborted.', error(110, 'Connection timed out'))
May 13 17:38:11 bumblebee pulp: nectar.downloaders.threaded:ERROR:
(17541-47008) Traceback (most recent call last):
May 13 17:38:11 bumblebee pulp: nectar.downloaders.threaded:ERROR:
(17541-47008) File
"/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line
190, in _fetch
May 13 17:38:11 bumblebee pulp: nectar.downloaders.threaded:ERROR:
(17541-47008) response = session.get(request.url, headers=headers)
May 13 17:38:11 bumblebee pulp: nectar.downloaders.threaded:ERROR:
(17541-47008) File
"/usr/lib/python2.7/site-packages/requests/sessions.py", line 469, in get
May 13 17:38:11 bumblebee pulp: nectar.downloaders.threaded:ERROR:
(17541-47008) return self.request('GET', url, **kwargs)
May 13 17:38:11 bumblebee pulp: nectar.downloaders.threaded:ERROR:
(17541-47008) File
"/usr/lib/python2.7/site-packages/requests/sessions.py", line 457, in
request
May 13 17:38:11 bumblebee pulp: nectar.downloaders.threaded:ERROR:
(17541-47008) resp = self.send(prep, **send_kwargs)
May 13 17:38:11 bumblebee pulp: nectar.downloaders.threaded:ERROR:
(17541-47008) File
"/usr/lib/python2.7/site-packages/requests/sessions.py", line 569, in send
May 13 17:38:11 bumblebee pulp: nectar.downloaders.threaded:ERROR:
(17541-47008) r = adapter.send(request, **kwargs)
May 13 17:38:11 bumblebee pulp: nectar.downloaders.threaded:ERROR:
(17541-47008) File
"/usr/lib/python2.7/site-packages/requests/adapters.py", line 407, in send
May 13 17:38:11 bumblebee pulp: nectar.downloaders.threaded:ERROR:
(17541-47008) raise ConnectionError(err, request=request)
May 13 17:38:11 bumblebee pulp: nectar.downloaders.threaded:ERROR:
(17541-47008) ConnectionError: ('Connection aborted.', error(110,
'Connection timed out'))
May 13 17:38:11 bumblebee pulp: pulp_rpm.plugins.importers.iso.sync:ERROR:
Failed to download
https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/iso/PULP_MANIFEST:
('Connection aborted.', error(110, 'Connection timed out')).
May 13 17:38:12 bumblebee pulp: celery.worker.job:ERROR: (17431-11936)
Task
pulp.server.managers.repo.sync.sync[0e0bbc76-43de-4dd4-b45c-edff18a59644]
raised unexpected: PulpExecutionException('Importer indicated a failed
response',)
May 13 17:38:12 bumblebee pulp: celery.worker.job:ERROR: (17431-11936)
Traceback (most recent call last):
May 13 17:38:12 bumblebee pulp: celery.worker.job:ERROR: (17431-11936)
File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in
trace_task
May 13 17:38:12 bumblebee pulp: celery.worker.job:ERROR: (17431-11936)
R = retval = fun(*args, **kwargs)
May 13 17:38:12 bumblebee pulp: celery.worker.job:ERROR: (17431-11936)
File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line
328, in __call__
May 13 17:38:12 bumblebee pulp: celery.worker.job:ERROR: (17431-11936)
return super(Task, self).__call__(*args, **kwargs)
May 13 17:38:12 bumblebee pulp: celery.worker.job:ERROR: (17431-11936)
File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in
__protected_call__
May 13 17:38:12 bumblebee pulp: celery.worker.job:ERROR: (17431-11936)
return self.run(*args, **kwargs)
May 13 17:38:12 bumblebee pulp: celery.worker.job:ERROR: (17431-11936)
File "/usr/lib/python2.7/site-packages/pulp/server/managers/repo/sync.py",
line 114, in sync
May 13 17:38:12 bumblebee pulp: celery.worker.job:ERROR: (17431-11936)
raise PulpExecutionException(_('Importer indicated a failed response'))
May 13 17:38:12 bumblebee pulp: celery.worker.job:ERROR: (17431-11936)
PulpExecutionException: Importer indicated a failed response
May 13 17:38:12 bumblebee pulp: celery.worker.job:INFO: Task
pulp.server.async.tasks._release_resource[09882136-6b6b-422d-bd55-3d312943cad3]
succeeded in 0.00555812800303s: None
==========================================================================
=========== successful curl command ======================================
[root at bumblebee ~]# curl -v \
--proxy $http_proxy \
https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/iso/PULP_MANIFEST
\
--insecure \
--cert /etc/pki/entitlement/$keyid.pem \
--key /etc/pki/entitlement/$keyid-key.pem
* About to connect() to proxy XXX.XXX.XXX.XXX port YY (#0)
* Trying XXX.XXX.XXX.XXX...
* Connected to XXX.XXX.XXX.XXX (XXX.XXX.XXX.XXX) port YY (#0)
* Establish HTTP proxy tunnel to cdn.redhat.com:443
* Proxy auth using Basic with user 'USER'
> CONNECT cdn.redhat.com:443 HTTP/1.1
> Host: cdn.redhat.com:443
> Proxy-Authorization: Basic XXXXXXXXXXXXXXXXXXXXXXXXXXXXX
> User-Agent: curl/7.29.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 Connection established
<
* Proxy replied OK to CONNECT request
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* NSS: client certificate from file
* subject: CN=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
* start date: Dec 09 05:00:00 2013 GMT
* expire date: Dec 09 04:59:59 2016 GMT
* common name: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
* issuer: E=ca-support at redhat.com,CN=Red Hat Candlepin Authority,OU=Red
Hat Network,O="Red Hat, Inc.",ST=North Carolina,C=US
* SSL connection using TLS_RSA_WITH_AES_256_CBC_SHA
* Server certificate:
* subject: CN=cdn.redhat.com,OU=Red Hat Network,O=Red
Hat,L=Raleigh,ST=North Carolina,C=US
* start date: May 14 19:48:02 2014 GMT
* expire date: May 11 19:48:02 2024 GMT
* common name: cdn.redhat.com
* issuer: E=ca-support at redhat.com,CN=Red Hat Entitlement Operations
Authority,OU=Red Hat Network,O="Red Hat, Inc.",ST=North Carolina,C=US
> GET /content/dist/rhel/server/7/7Server/x86_64/iso/PULP_MANIFEST HTTP/1.1
> User-Agent: curl/7.29.0
> Host: cdn.redhat.com
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: Apache
< ETag: "dde1e965a48e56cba8aac0b597f0c7a6:1426504103"
< Last-Modified: Mon, 16 Mar 2015 11:08:16 GMT
< Accept-Ranges: bytes
< Content-Length: 424
< Content-Type: text/plain
< X-Akamai-Request-ID: 5c57dff5.19737949
< Date: Wed, 13 May 2015 07:14:12 GMT
< X-Cache: TCP_MISS from a104-71-131-23.deploy.akamaitechnologies.com
(AkamaiGHost/7.2.0-15182023) (-)
< Connection: keep-alive
< EJ-HOST: edgejavaapp5.prod.a4.vary.redhat.com
<
RedHat-PassSync-1.1.6-i386.msi,0cf9b1e6998f756effc051a193c764e6f9976ac00b506db1381f8649404ec534,3059712
RedHat-PassSync-1.1.6-x86_64.msi,5430ac6ccb26340db4da82235e0bbd70c4a21f7ad7e7cd6d953159ea2f1c80df,3399680
rhel-server-7.1-x86_64-boot.iso,b6c57194130025cecc036676d43235508f4031d99041051ea1c61e259bd95903,358612992
rhel-server-7.1-x86_64-dvd.iso,3685468ec6cdcb70dfc85ebbc164da427dc2d762644c3c2ee1520f4f661c15ce,3890216960
* Connection #0 to host 10.226.8.34 left intact
[root at bumblebee ~]#
==========================================================================
=========== pulp rpm versions ============================================
[root at bumblebee conf.d]# rpm -qa | grep pulp
pulp-puppet-admin-extensions-2.6.1-1.el7.noarch
python-pulp-client-lib-2.6.1-1.el7.noarch
pulp-selinux-2.6.1-1.el7.noarch
pulp-rpm-plugins-2.6.1-1.el7.noarch
python-isodate-0.5.0-4.pulp.el7.noarch
pulp-server-2.6.1-1.el7.noarch
python-pulp-rpm-common-2.6.1-1.el7.noarch
pulp-rpm-admin-extensions-2.6.1-1.el7.noarch
python-kombu-3.0.24-5.pulp.el7.noarch
python-pulp-puppet-common-2.6.1-1.el7.noarch
pulp-puppet-plugins-2.6.1-1.el7.noarch
pulp-admin-client-2.6.1-1.el7.noarch
python-pulp-common-2.6.1-1.el7.noarch
python-pulp-bindings-2.6.1-1.el7.noarch
==========================================================================
More information about the Pulp-list
mailing list