[Pulp-list] Pulp sync failure - not well-formed (invalid token): line 44436, column 0

Webb, Reece Reece.Webb at ucsf.edu
Mon Mar 23 17:52:21 UTC 2015 

Forgot to include version info.. I had to upgrade to 2.6 to get things working..

# rpm -qa | grep pulp
python-isodate-0.5.0-4.pulp.el7.noarch
pulp-puppet-admin-extensions-2.6.0-0.8.rc.el7.noarch
pulp-admin-client-2.6.0-0.10.rc.el7.noarch
python-pulp-rpm-common-2.6.0-0.8.rc.el7.noarch
pulp-puppet-plugins-2.6.0-0.8.rc.el7.noarch
python-kombu-3.0.24-5.pulp.el7.noarch
pulp-rpm-admin-extensions-2.6.0-0.8.rc.el7.noarch
python-pulp-common-2.6.0-0.10.rc.el7.noarch
python-pulp-client-lib-2.6.0-0.10.rc.el7.noarch
pulp-server-2.6.0-0.10.rc.el7.noarch
python-pulp-puppet-common-2.6.0-0.8.rc.el7.noarch
pulp-rpm-plugins-2.6.0-0.8.rc.el7.noarch
python-pulp-bindings-2.6.0-0.10.rc.el7.noarch
pulp-selinux-2.6.0-0.10.rc.el7.noarch





On 3/23/15, 10:50 AM, "Webb, Reece" <Reece.Webb at ucsf.edu> wrote:

>I’ve been having failures syncing with certain RH repositories as well. For me, it’s the 7Workstation and 5Client repo’s (7Server doesn’t have any problems). I get a “Forbidden”, which would seem to indicate a problem with the cert… but this sync problem is intermittent. If I manually kick off a sync, sometimes it works, often it doesn’t.
>
>Mar 23 10:24:07 yummy pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[f8759f66-7416-4fd1-9ee4-6005ca06df8a] succeeded in 0.0339540460263s: None
>Mar 23 10:24:07 yummy pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading metadata from https://cdn.redhat.com/content/dist/rhel/workstation/7/7Workstation/x86_64/kickstart/.
>Mar 23 10:24:07 yummy pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): cdn.redhat.com
>Mar 23 10:24:08 yummy pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (1986-41632) sync failed
>Mar 23 10:24:08 yummy pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (1986-41632) Traceback (most recent call last):
>Mar 23 10:24:08 yummy pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (1986-41632)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 104, in run
>Mar 23 10:24:08 yummy pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (1986-41632)     metadata_files = self.get_metadata()
>Mar 23 10:24:08 yummy pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (1986-41632)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 198, in get_metadata
>Mar 23 10:24:08 yummy pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (1986-41632)     raise FailedException(str(e))
>Mar 23 10:24:08 yummy pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (1986-41632) FailedException: Forbidden
>Mar 23 10:24:08 yummy pulp: celery.worker.job:ERROR: (1595-41632) Task pulp.server.managers.repo.sync.sync[80a6866e-0ed6-456a-8b7a-8d0cadbd84aa] raised unexpected: PulpExecutionException('Importer indicated a failed response',)
>
>
>
>Here’s two back-to-back sync’s, first failed (messages above), second was fine.
>
># pulp-admin rpm repo sync run --repo-id=base-7Workstation
>+----------------------------------------------------------------------+
>              Synchronizing Repository [base-7Workstation]
>+----------------------------------------------------------------------+
>
>This command may be exited via ctrl+c without affecting the request.
>
>
>Downloading metadata...
>[-]
>... failed
>
>Forbidden
>
>
>Task Failed
>
>Importer indicated a failed response
>
>
># pulp-admin rpm repo sync run --repo-id=base-7Workstation
>+----------------------------------------------------------------------+
>              Synchronizing Repository [base-7Workstation]
>+----------------------------------------------------------------------+
>
>This command may be exited via ctrl+c without affecting the request.
>
>
>Downloading metadata...
>[/]
>... completed
>
>Downloading repository content...
>[==================================================] 100%
>RPMs:       0/0 items
>Delta RPMs: 0/0 items
>
>... completed
>
>Downloading distribution files...
>[==================================================] 100%
>Distributions: 0/0 items
>... completed
>
>Importing errata...
>[-]
>... completed
>
>Importing package groups/categories...
>[-]
>... completed
>
>Task Succeeded
>
>Copying files
>[\]
>... completed
>
>Initializing repo metadata
>[-]
>... completed
>
>Publishing Distribution files
>[-]
>... completed
>
>Publishing RPMs
>[-]
>... completed
>
>Publishing Delta RPMs
>... skipped
>
>Publishing Errata
>[-]
>... completed
>
>Publishing Comps file
>[==================================================] 100%
>86 of 86 items
>... completed
>
>Publishing Metadata.
>[-]
>... completed
>
>Closing repo metadata
>[-]
>... completed
>
>Generating sqlite files
>[|]
>... completed
>
>Publishing files to web
>[|]
>... completed
>
>Writing Listings File
>[-]
>... completed
>
>Writing Listings File
>[\]
>... completed
>
>
>Task Succeeded
>
>
>—-
>
>The other issue I have is that I’m syncing the kickstart trees…. When I first setup Pulp, it was syncing 7.1 server and workstation. Early last week, my workstation repo looks to have fallen back to 7.0. Looking at the local workstation treeinfo file, it lists version=7.0, whereas my server treeinfo lists 7.1.
>
>I used curl to see what’s on the RH repo, treeinfo for both workstation and server list 7.0. (I disabled my server sync so that it remains at 7.1)
>
>These are the two feeds I’m syncing against:
>
>https://cdn.redhat.com/content/dist/rhel/workstation/7/7Workstation/x86_64/kickstart/
>https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/kickstart/
>
>
>
>It’s weird, if I look at the redhat-release package, it’s 7.1:
>
># ls -al /var/lib/pulp/published/yum/http/repos/base-7Workstation/redhat-release-workstation-7.1-1.el7.x86_64.rpm
>
>
>But the images/pxeboot files are all from 7.0
>
># ls -al /var/lib/pulp/content/distribution/ks-Red\ Hat\ Enterprise\ Linux-Workstation-7.0-x86_64/images/pxeboot/
>total 72456
>drwxr-xr-x. 2 apache apache       55 Mar 14 14:44 .
>drwxr-xr-x. 3 apache apache       20 Mar 14 14:44 ..
>-rw-r--r--. 1 apache apache 35544564 Mar 14 14:44 initrd.img
>-rw-r--r--. 1 apache apache 33744152 Mar 14 14:44 upgrade.img
>-rw-r--r--. 1 apache apache  4902000 Mar 14 14:44 vmlinuz
>
>
>There is no longer a 7.1 workstation distribution directory.
>
># ls -al /var/lib/pulp/content/distribution/
>total 8
>drwxr-xr-x. 5 apache apache 4096 Mar 18 08:46 .
>drwxr-xr-x. 6 apache apache   74 Mar 14 14:36 ..
>drwxrwxr-x. 5 apache apache 4096 Mar 16 14:31 ks-Red Hat Enterprise Linux Client-5.11-x86_64
>drwxrwxr-x. 5 apache apache   60 Mar 14 22:30 ks-Red Hat Enterprise Linux-Server-7.1-x86_64
>drwxrwxr-x. 4 apache apache   47 Mar 14 14:44 ks-Red Hat Enterprise Linux-Workstation-7.0-x86_64
>
>
>
>I’ve only been using Pulp for 2 weeks, so I’m learning as I go. The above doesn’t seem right, but maybe I don’t have something configured correctly?
>
>Thanks for any help,
>Reece
>
>
>
>
>
>
>On 3/23/15, 7:39 AM, "Paul Jochum" <paul.jochum at alcatel-lucent.com> wrote:
>
>>Hi Barnaby:
>>
>>$ rpm -qa | grep pulp
>>python-pulp-rpm-common-2.5.3-1.el6.noarch
>>pulp-server-2.5.3-1.el6.noarch
>>pulp-puppet-plugins-2.5.3-1.el6.noarch
>>pulp-selinux-2.5.3-1.el6.noarch
>>python-isodate-0.5.0-4.pulp.el6.noarch
>>python-kombu-3.0.15-12.pulp.el6.noarch
>>python-pulp-puppet-common-2.5.3-1.el6.noarch
>>python-pulp-bindings-2.5.3-1.el6.noarch
>>pulp-admin-client-2.5.3-1.el6.noarch
>>mod_wsgi-3.4-2.pulp.el6.x86_64
>>pulp-rpm-plugins-2.5.3-1.el6.noarch
>>pulp-puppet-admin-extensions-2.5.3-1.el6.noarch
>>python-pulp-common-2.5.3-1.el6.noarch
>>python-pulp-client-lib-2.5.3-1.el6.noarch
>>pulp-rpm-admin-extensions-2.5.3-1.el6.noarch
>>
>>thanks,
>>Paul
>>
>>
>>On 03/23/2015 09:25 AM, Barnaby Court wrote:
>>> Hi Paul, What version of Pulp are you running? "rpm -qa | grep pulp " should give all the pertinent information.  Thanks!
>>>
>>> -Barnaby
>>>
>>> ----- Original Message -----
>>> From: "Paul Jochum" <paul.jochum at alcatel-lucent.com>
>>> To: pulp-list at redhat.com
>>> Sent: Monday, March 23, 2015 8:50:48 AM
>>> Subject: [Pulp-list] Pulp sync failure - not well-formed (invalid token): line 44436, column 0
>>>
>>> Hi All:
>>>
>>>       For the last 2 weeks, one of my daily pulp sync's has been
>>> failing.  It is always the same repo (9 other syncs on that server run
>>> fine).  All of the sync's are to Red Hat, for different RHEL 7
>>> repositories.  The one that is failing is for rhel-x86_64-server-7-source.
>>>
>>> This is what I see for the failing sync:
>>> *****************************************************************************
>>>
>>> $ pulp-admin -u admin -p admin rpm repo sync run --repo-id=$i
>>> +----------------------------------------------------------------------+
>>>            Synchronizing Repository [rhel-x86_64-server-7-source]
>>> +----------------------------------------------------------------------+
>>>
>>> This command may be exited via ctrl+c without affecting the request.
>>>
>>>
>>> Downloading metadata...
>>> [\]
>>> ... failed
>>>
>>> not well-formed (invalid token): line 44436, column 0
>>>
>>>
>>> Task Failed
>>>
>>> Importer indicated a failed response
>>>
>>> *****************************************************************************
>>> And this is in /var/log/messages:
>>> Mar 23 07:43:39 lss-pulp02 pulp: celery.worker.strategy:INFO: Received
>>> task:
>>> pulp.server.async.tasks._reserve_resource[94539898-05e5-4800-aa61-b75aa05a38c5]
>>> Mar 23 07:43:39 lss-pulp02 pulp: celery.worker.job:INFO: Task
>>> pulp.server.async.tasks._reserve_resource[94539898-05e5-4800-aa61-b75aa05a38c5]
>>> succeeded in 0.0402820348972s:
>>> u'reserved_resource_worker-1 at lss-pulp02.ih.lucent.com.dq'
>>> Mar 23 07:43:39 lss-pulp02 pulp: celery.worker.strategy:INFO: Received
>>> task:
>>> pulp.server.tasks.repository.sync_with_auto_publish[e8fe2c7a-ee33-4548-bbf4-176e2018a997]
>>> Mar 23 07:43:39 lss-pulp02 pulp:
>>> pulp_rpm.plugins.importers.yum.sync:INFO: Downloading metadata from
>>> https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/source/SRPMS/.
>>> Mar 23 07:43:39 lss-pulp02 pulp:
>>> requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS
>>> connection (1): cdn.redhat.com
>>> Mar 23 07:43:40 lss-pulp02 pulp:
>>> pulp_rpm.plugins.importers.yum.sync:INFO: Parsing metadata.
>>> Mar 23 07:43:40 lss-pulp02 pulp:
>>> pulp_rpm.plugins.importers.yum.sync:INFO: Downloading metadata files.
>>> Mar 23 07:43:40 lss-pulp02 pulp:
>>> requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS
>>> connection (1): cdn.redhat.com
>>> Mar 23 07:43:40 lss-pulp02 pulp:
>>> requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS
>>> connection (1): cdn.redhat.com
>>> Mar 23 07:43:40 lss-pulp02 pulp:
>>> requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS
>>> connection (1): cdn.redhat.com
>>> Mar 23 07:43:40 lss-pulp02 pulp:
>>> requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS
>>> connection (1): cdn.redhat.com
>>> Mar 23 07:43:42 lss-pulp02 pulp:
>>> pulp_rpm.plugins.importers.yum.sync:INFO: Generating metadata databases.
>>> Mar 23 07:43:45 lss-pulp02 pulp:
>>> pulp_rpm.plugins.importers.yum.sync:ERROR: sync failed
>>> Mar 23 07:43:45 lss-pulp02 pulp:
>>> pulp_rpm.plugins.importers.yum.sync:ERROR: Traceback (most recent call
>>> last):
>>> Mar 23 07:43:45 lss-pulp02 pulp:
>>> pulp_rpm.plugins.importers.yum.sync:ERROR:   File
>>> "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/sync.py",
>>> line 105, in run
>>> Mar 23 07:43:45 lss-pulp02 pulp:
>>> pulp_rpm.plugins.importers.yum.sync:ERROR:     metadata_files =
>>> self.get_metadata()
>>> Mar 23 07:43:45 lss-pulp02 pulp:
>>> pulp_rpm.plugins.importers.yum.sync:ERROR:   File
>>> "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/sync.py",
>>> line 212, in get_metadata
>>> Mar 23 07:43:45 lss-pulp02 pulp:
>>> pulp_rpm.plugins.importers.yum.sync:ERROR: metadata_files.generate_dbs()
>>> Mar 23 07:43:45 lss-pulp02 pulp:
>>> pulp_rpm.plugins.importers.yum.sync:ERROR:   File
>>> "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/repomd/metadata.py",
>>> line 275, in generate_dbs
>>> Mar 23 07:43:45 lss-pulp02 pulp:
>>> pulp_rpm.plugins.importers.yum.sync:ERROR:     for element in generator:
>>> Mar 23 07:43:45 lss-pulp02 pulp:
>>> pulp_rpm.plugins.importers.yum.sync:ERROR:   File
>>> "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/repomd/packages.py",
>>> line 53, in package_list_generator
>>> Mar 23 07:43:45 lss-pulp02 pulp:
>>> pulp_rpm.plugins.importers.yum.sync:ERROR:     for event, element in
>>> xml_iterator:
>>> Mar 23 07:43:45 lss-pulp02 pulp:
>>> pulp_rpm.plugins.importers.yum.sync:ERROR:   File "<string>", line 64,
>>> in __iter__
>>> Mar 23 07:43:45 lss-pulp02 pulp:
>>> pulp_rpm.plugins.importers.yum.sync:ERROR: SyntaxError: not well-formed
>>> (invalid token): line 44436, column 0
>>> Mar 23 07:43:45 lss-pulp02 pulp: celery.worker.job:ERROR: Task
>>> pulp.server.tasks.repository.sync_with_auto_publish[e8fe2c7a-ee33-4548-bbf4-176e2018a997]
>>> raised unexpected: PulpExecutionException('Importer indicated a failed
>>> response',)
>>> Mar 23 07:43:45 lss-pulp02 pulp: celery.worker.job:ERROR: Traceback
>>> (most recent call last):
>>> Mar 23 07:43:45 lss-pulp02 pulp: celery.worker.job:ERROR:   File
>>> "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 240, in
>>> trace_task
>>> Mar 23 07:43:45 lss-pulp02 pulp: celery.worker.job:ERROR:     R = retval
>>> = fun(*args, **kwargs)
>>> Mar 23 07:43:45 lss-pulp02 pulp: celery.worker.job:ERROR:   File
>>> "/usr/lib/python2.6/site-packages/pulp/server/async/tasks.py", line 316,
>>> in __call__
>>> Mar 23 07:43:45 lss-pulp02 pulp: celery.worker.job:ERROR:     return
>>> super(Task, self).__call__(*args, **kwargs)
>>> Mar 23 07:43:45 lss-pulp02 pulp: celery.worker.job:ERROR:   File
>>> "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 437, in
>>> __protected_call__
>>> Mar 23 07:43:45 lss-pulp02 pulp: celery.worker.job:ERROR:     return
>>> self.run(*args, **kwargs)
>>> Mar 23 07:43:45 lss-pulp02 pulp: celery.worker.job:ERROR:   File
>>> "/usr/lib/python2.6/site-packages/pulp/server/tasks/repository.py", line
>>> 210, in sync_with_auto_publish
>>> Mar 23 07:43:45 lss-pulp02 pulp: celery.worker.job:ERROR: sync_result =
>>> managers.repo_sync_manager().sync(repo_id, sync_config_override=overrides)
>>> Mar 23 07:43:45 lss-pulp02 pulp: celery.worker.job:ERROR:   File
>>> "/usr/lib/python2.6/site-packages/pulp/server/managers/repo/sync.py",
>>> line 113, in sync
>>> Mar 23 07:43:45 lss-pulp02 pulp: celery.worker.job:ERROR:     raise
>>> PulpExecutionException(_('Importer indicated a failed response'))
>>> Mar 23 07:43:45 lss-pulp02 pulp: celery.worker.job:ERROR:
>>> PulpExecutionException: Importer indicated a failed response
>>>
>>> *****************************************************************************
>>>
>>> Any suggestions on how to clean it up?  Would any additional information
>>> help?
>>>
>>> thanks,
>>> Paul
>>>
>>> _______________________________________________
>>> Pulp-list mailing list
>>> Pulp-list at redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-list
>>
>>_______________________________________________
>>Pulp-list mailing list
>>Pulp-list at redhat.com
>>https://www.redhat.com/mailman/listinfo/pulp-list

More information about the Pulp-list mailing list