[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [Pulp-list] pypi sync no longer working after pypi.org migration with pulp 2.14 and pulp-python-plugins 2.0.2-1



Dustin, 

Thanks for the heads up. We're looking into it now and will have a fix soon.

David


On Wed, Jul 25, 2018 at 1:29 PM Dustin McNabb <dustin mcnabb gmail com> wrote:
Apparently this essential pypi.org fix at the github link below was left out of the pulp-python-2.0.3-0.1 release for the pulp 2.16.4 beta. It is essential for pypi.org support as path is no longer an option and the url formation logic in models.py no longer applies. I have opened another issue at https://pulp.plan.io/issues/3884 

https://github.com/pulp/pulp_python/commit/79b46411a049e63fc24ba97a60b51fe0574b268c

Dustin McNabb



On Jul 25, 2018, at 7:32 AM, Brian Bouterse <bbouters redhat com> wrote:

Thank you for investigating and reporting this. We are going to try to reproduce and patch the version of Celery we distribute with Pulp.

On Wed, Jul 25, 2018 at 10:14 AM, Dustin McNabb <dustin mcnabb gmail com> wrote:
Sure: https://pulp.plan.io/issues/3882

Dustin McNabb




On Jul 25, 2018, at 6:55 AM, Bihan Zhang <bizhang redhat com> wrote:

Dustin,
Would you mind opening a pulp issue with a description of the issue and the patch? https://pulp.plan.io/issues/new 
I'm not too familiar with the tasking system, and it would be good for the issue to go through the regular triage process.


Cheers,
Bihan

On Wed, Jul 25, 2018 at 8:18 AM, Dustin McNabb <dustin mcnabb gmail com> wrote:
Bihan -

Thanks for the fast response. However, I’ve upgraded my lab from pulp 2.14 to this beta and it introduces a new problem by upgrading to celery 4.0.2-4. Specifically, we now have this issue https://github.com/celery/celery/issues/3620. It was fixed in celery 4.1. You need multiple tasks in the queue upon initial startup to reproduce this issue. You can also see more discussion about this issue here: https://github.com/celery/celery/pull/3752 and here https://github.com/celery/celery/issues/4109#issuecomment-314367686. I implemented this fix in my lab environment and it resolved my issue.

# Problem with pulp_resource_manager service
Jul 25 06:10:03 pulp-lab pulp[17698]: celery.worker:CRITICAL: (17698-58784)   File "/usr/lib64/python2.7/site-packages/billiard/pool.py", line 1486, in apply_async
Jul 25 06:10:03 pulp-lab  pulp[17698]: celery.worker:CRITICAL: (17698-58784)     self._quick_put((TASK, (result._job, None, func, args, kwds)))
Jul 25 06:10:03 pulp-lab pulp[17698]: celery.worker:CRITICAL: (17698-58784) TypeError: 'NoneType' object is not callable

# Here is the fix that was ultimately implemented which moves the consumption step below the object registration in worker/loops.py.
-    consumer.consume()
-    obj.on_ready()
     obj.controller.register_with_event_loop(hub)
     obj.register_with_event_loop(hub)
+    consumer.consume()
+    obj.on_ready()


Thanks
Dustin McNabb




On Jul 24, 2018, at 2:23 PM, Bihan Zhang <bizhang redhat com> wrote:

Dustin,

These fixes were implemented[0] and were just release in pulp 2.16.4 beta and pulp_python 2.0.3 beta today.
Here's our release schedule in case you want to keep an eye out on when this is going to go GA: https://pulp.plan.io/projects/pulp/wiki/2164_Release_Status



On Tue, Jul 24, 2018 at 4:50 PM, Dustin McNabb <dustin mcnabb gmail com> wrote:
I have pulp-server 2.14.3-1 and pulp-python-plugins 2.0.2-1 and I can no longer sync form python.pypi.org. I realize they switched to pypi.org and files.pythonhosted.org, but the old site still redirects requests. However, the URLs where pulp-python-plugins expects to find packages no longer exist. Also, the pulp-python packages have not been updated since 2017. Does this mean that we don’t have support for the new pypi.org warehouse using pulp2? Or is there a legacy feed URL I can use?

Jul 24 20:33:55 pulp-test-server pulp: nectar.downloaders.threaded:INFO: Download failed: Download of https://pypi.org/simple/pypi/tinydb/json/ failed with code 404: Not Found
Jul 24 20:33:55 pulp-test-server pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (37): pypi.org
Jul 24 20:33:55 pulp-test-server pulp: nectar.downloaders.threaded:INFO: Download failed: Download of https://pypi.org/simple/pypi/setuptools/json/ failed with code 404: Not Found
Jul 24 20:33:55 pulp-test-server pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (38): pypi.org
Jul 24 20:33:55 pulp-test-server pulp: nectar.downloaders.threaded:INFO: Download failed: Download of https://pypi.org/simple/pypi/urllib3/json/ failed with code 404: Not Found
Jul 24 20:33:55 pulp-test-server pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (39): pypi.org
Jul 24 20:33:55 pulp-test-server pulp: nectar.downloaders.threaded:INFO: Download failed: Download of https://pypi.org/simple/pypi/chardet/json/ failed with code 404: Not Found
Jul 24 20:33:55 pulp-test-server pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (40): pypi.org
Jul 24 20:33:55 pulp-test-server pulp: nectar.downloaders.threaded:INFO: Download failed: Download of https://pypi.org/simple/pypi/passlib/json/ failed with code 404: Not Found
Jul 24 20:33:55 pulp-test-server pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (41): pypi.org
Jul 24 20:33:55 pulp-test-server pulp: nectar.downloaders.threaded:INFO: Download failed: Download of https://pypi.org/simple/pypi/setuptools/json/ failed with code 404: Not Found
Jul 24 20:33:55 pulp-test-server pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (42): pypi.org
Jul 24 20:33:55 pulp-test-server pulp: nectar.downloaders.threaded:INFO: Download failed: Download of https://pypi.org/simple/pypi/gunicorn/json/ failed with code 404: Not Found
Jul 24 20:33:55 pulp-test-server pulp: nectar.downloaders.threaded:INFO: Download failed: Download of https://pypi.org/simple/pypi/idna/json/ failed with code 404: Not Found

Thanks

Dustin McNabb





_______________________________________________
Pulp-list mailing list
Pulp-list redhat com
https://www.redhat.com/mailman/listinfo/pulp-list





_______________________________________________
Pulp-list mailing list
Pulp-list redhat com
https://www.redhat.com/mailman/listinfo/pulp-list


_______________________________________________
Pulp-list mailing list
Pulp-list redhat com
https://www.redhat.com/mailman/listinfo/pulp-list

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]