[Pulp-list] Pulp 3 internal error
Bin Li (BLOOMBERG/ 120 PARK)
bli111 at bloomberg.net
Tue Jun 25 20:10:53 UTC 2019
This does give me the metadata. I was also able to get package by specifying the rpm explicitly. Thanks for all your help.
# yum install http://localhost:24816/pulp/content/bloomberg/systems_third_party_rhel7.2/admin-python-36-3.6.5-1.el7.x86_64.rpm
Loaded plugins: product-id, search-disabled-repos, subscription-manager
hp-scripting-tools-10.10-53.rhel7.x86_64.rpm | 3.7 MB 00:00:00
Examining /var/tmp/yum-root-FSly5B/hp-scripting-tools-10.10-53.rhel7.x86_64.rpm: hp-scripting-tools-10.10-53.rhel7.x86_64
Marking /var/tmp/yum-root-FSly5B/hp-scripting-tools-10.10-53.rhel7.x86_64.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package hp-scripting-tools.x86_64 0:10.10-53.rhel7 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
========================================================================================================================
Package Arch Version Repository Size
========================================================================================================================
Installing:
hp-scripting-tools x86_64 10.10-53.rhel7 /hp-scripting-tools-10.10-53.rhel7.x86_64 12 M
Transaction Summary
========================================================================================================================
Install 1 Package
# http get http://localhost:24816/pulp/content/bloomberg/systems_third_party_rhel7.2/repodata/repomd.xml
HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Length: 3552
Content-Type: text/xml
Date: Tue, 25 Jun 2019 19:51:45 GMT
Last-Modified: Tue, 25 Jun 2019 14:59:31 GMT
Server: Python/3.6 aiohttp/3.5.4
<?xml version="1.0" encoding="UTF-8"?>
<repomd xmlns="http://linux.duke.edu/metadata/repo" xmlns:rpm="http://linux.duke.edu/metadata/rpm">
<revision>1561474770</revision>
<data type="primary">
<checksum type="sha256">851fae6a840f9d579db6299b0af4f6f79c0d110524ffae10f5beac4f070611dc</checksum>
<open-checksum type="sha256">05d6b1d9e534737479258ca6860225ade087b76cfa0dbb411937f9ab21ce235a</open-checksum>
<location href="repodata/851fae6a840f9d579db6299b0af4f6f79c0d110524ffae10f5beac4f070611dc-primary.xml.gz"/>
<timestamp>1561474769</timestamp>
<size>85850</size>
<open-size>1126399</open-size>
</data>
<data type="filelists">
<checksum type="sha256">060b8895700bc9ea458660398482d1e4de0feb07327cda5411ed98883de6597f</checksum>
<open-checksum type="sha256">46c2f0fe79e52f23f2239ea4e377c3e51f7193c810482e4c104586b8bacdb429</open-checksum>
<location href="repodata/060b8895700bc9ea458660398482d1e4de0feb07327cda5411ed98883de6597f-filelists.xml.gz"/>
<timestamp>1561474769</timestamp>
<size>458428</size>
<open-size>11996064</open-size>
From: dkliban at redhat.com At: 06/25/19 15:42:27To: Bin Li (BLOOMBERG/ 120 PARK )
Cc: brocha at redhat.com, pulp-list at redhat.com
Subject: Re: [Pulp-list] Pulp 3 internal error
The RPM plugin does not list directories right now. Feel free to file an issue about this.
What happens when you GET http://localhost:24816/pulp/content/bloomberg/systems_third_party_rhel7.2/repodata/repomd.xml ?
On Tue, Jun 25, 2019 at 3:35 PM Bin Li (BLOOMBERG/ 120 PARK) <bli111 at bloomberg.net> wrote:
Updated CONTENT_HOST = 'http://localhost:24816' in settings.py, restarted all daemons and recreated distribution. Now I am getting 404 error:
# http get http://localhost:24816/pulp/content/bloomberg/systems_third_party_rhel7.2/
HTTP/1.1 404 Not Found
Content-Length: 14
Content-Type: text/plain; charset=utf-8
Date: Tue, 25 Jun 2019 19:14:28 GMT
Server: Python/3.6 aiohttp/3.5.4
404: Not Found
The following is in the log:
Jun 25 15:19:32 ip-1-76-158-221 gunicorn[180694]: 127.0.0.1 - admin [25/Jun/2019:19:19:32 +0000] "GET /pulp/api/v3/publications/rpm/rpm/ HTTP/1.1" 200 567 "-" "HTTPie/0.9.4"
Jun 25 15:19:41 ip-1-76-158-221 rc.bloomberg[1549]: pid 149131's current scheduling policy: SCHED_OTHER pid 149131's current scheduling priority: 0
Jun 25 15:19:41 ip-1-76-158-221 rc.bloomberg[1549]: chrt: failed to set pid 149131's policy: Operation not permitted
Jun 25 15:19:48 ip-1-76-158-221 gunicorn[180694]: 127.0.0.1 - admin [25/Jun/2019:19:19:48 +0000] "GET /pulp/api/v3/repositories/ HTTP/1.1" 200 1353 "-" "HTTPie/0.9.4"
Jun 25 15:19:59 ip-1-76-158-221 gunicorn[180694]: 127.0.0.1 - admin [25/Jun/2019:19:19:59 +0000] "GET /pulp/api/v3/repositories/ HTTP/1.1" 200 1353 "-" "HTTPie/0.9.4"
Jun 25 15:20:00 ip-1-76-158-221 gunicorn[180694]: 127.0.0.1 - admin [25/Jun/2019:19:20:00 +0000] "GET /pulp/api/v3/remotes/rpm/rpm/ HTTP/1.1" 200 2383 "-" "HTTPie/0.9.4"
Jun 25 15:20:01 ip-1-76-158-221 gunicorn[180694]: 127.0.0.1 - admin [25/Jun/2019:19:20:01 +0000] "GET /pulp/api/v3/content/rpm/packages/ HTTP/1.1" 200 1873703 "-" "HTTPie/0.9.4"
The distribution is:
{
"count": 1,
"next": null,
"previous": null,
"results": [
{
"_created": "2019-06-25T19:13:54.154550Z",
"_href": "/pulp/api/v3/distributions/rpm/rpm/f3117050-9373-479d-ae14-63ac2f37f30c/",
"base_path": "bloomberg/systems_third_party_rhel7.2",
"base_url": "http://localhost:24816/pulp/content/bloomberg/systems_third_party_rhel7.2",
"content_guard": null,
"name": "systems_third_party_rhel7.2",
"publication": "/pulp/api/v3/publications/rpm/rpm/894a8109-001b-42fc-8658-4ac28bc52c0e/"
}
]
}
Thanks
From: brocha at redhat.com At: 06/25/19 15:09:04To: Bin Li (BLOOMBERG/ 120 PARK )
Cc: pulp-list at redhat.com
Subject: Re: [Pulp-list] Pulp 3 internal error
The port for content host recently changed to 24816
CONTENT_HOST = 'http://localhost:24816'
Bruno Rocha
Senior Quality Engineer - Cloud Management QE - Pulp Project
irc: rochacbruno
social: http://about.me/rochacbruno
“Progress is the realization of utopia.”
On Tue, Jun 25, 2019 at 4:01 PM Bin Li (BLOOMBERG/ 120 PARK) <bli111 at bloomberg.net> wrote:
Adding CONTENT_HOST = 'http://localhost:8080' to setting.py fixed the issue. However, got connection error when try to access a distribution.
# http get http://localhost:8080/pulp/content/bloomberg/systems_third_party_rhel7.2/
http: error: ConnectionError: ('Connection aborted.', error(111, 'Connection refused')) while doing GET request to URL: http://localhost:8080/pulp/content/bloomberg/systems_third_party_rhel7.2/
The port 8080 is not listening. What am I doing wrong?
The distribution is as following:
{
"count": 1,
"next": null,
"previous": null,
"results": [
{
"_created": "2019-06-25T18:51:14.803243Z",
"_href": "/pulp/api/v3/distributions/rpm/rpm/6cde0eda-ea68-4d22-920d-c62173c4d80e/",
"base_path": "bloomberg/systems_third_party_rhel7.2",
"base_url": "http://localhost:8080/pulp/content/bloomberg/systems_third_party_rhel7.2",
"content_guard": null,
"name": "systems_third_party_rhel7.2",
"publication": "/pulp/api/v3/publications/rpm/rpm/894a8109-001b-42fc-8658-4ac28bc52c0e/"
}
]
}
Thanks
From: dkliban at redhat.com At: 06/25/19 14:07:56To: Bin Li (BLOOMBERG/ 120 PARK )
Cc: pulp-list at redhat.com
Subject: Re: [Pulp-list] Pulp 3 internal error
This is the same bug as I mentioned earlier. You need to set either the PULP_CONTENT_HOST environment variable or CONTENT_HOST setting in /etc/pulp/settings.py
On Tue, Jun 25, 2019 at 1:57 PM Bin Li (BLOOMBERG/ 120 PARK) <bli111 at bloomberg.net> wrote:
Here is the traceback from the log:
Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: pulp: django.request:ERROR: Internal Server Error: /pulp/api/v3/tasks/
Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: Traceback (most recent call last):
Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File "/opt/python/3.6.5/lib/python3.6/site-packages/django/core/handlers/exception.py", line 34, in inner
Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: response = get_response(request)
Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File "/opt/python/3.6.5/lib/python3.6/site-packages/django/core/handlers/base.py", line 115, in _get_response
Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: response = self.process_exception_by_middleware(e, request)
Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File "/opt/python/3.6.5/lib/python3.6/site-packages/django/core/handlers/base.py", line 113, in _get_response
Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: response = wrapped_callback(request, *callback_args, **callback_kwargs)
Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File "/opt/python/3.6.5/lib/python3.6/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: return view_func(*args, **kwargs)
Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File "/opt/python/3.6.5/lib/python3.6/site-packages/rest_framework/viewsets.py", line 116, in view
Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: return self.dispatch(request, *args, **kwargs)
Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File "/opt/python/3.6.5/lib/python3.6/site-packages/rest_framework/views.py", line 495, in dispatch
Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: response = self.handle_exception(exc)
Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File "/opt/python/3.6.5/lib/python3.6/site-packages/rest_framework/views.py", line 455, in handle_exception
Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: self.raise_uncaught_exception(exc)
Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File "/opt/python/3.6.5/lib/python3.6/site-packages/rest_framework/views.py", line 492, in dispatch
Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: response = handler(request, *args, **kwargs)
Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File "/opt/python/3.6.5/lib/python3.6/site-packages/rest_framework/mixins.py", line 45, in list
Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: return self.get_paginated_response(serializer.data)
Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File "/opt/python/3.6.5/lib/python3.6/site-packages/rest_framework/serializers.py", line 768, in data
Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: ret = super(ListSerializer, self).data
Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File "/opt/python/3.6.5/lib/python3.6/site-packages/rest_framework/serializers.py", line 262, in data
Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: self._data = self.to_representation(self.instance)
Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File "/opt/python/3.6.5/lib/python3.6/site-packages/rest_framework/serializers.py", line 686, in to_representation
Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: self.child.to_representation(item) for item in iterable
Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File "/opt/python/3.6.5/lib/python3.6/site-packages/rest_framework/serializers.py", line 686, in <listcomp>
Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: self.child.to_representation(item) for item in iterable
Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File "/opt/python/3.6.5/lib/python3.6/site-packages/rest_framework/serializers.py", line 530, in to_representation
Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: ret[field.field_name] = field.to_representation(attribute)
Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File "/opt/python/3.6.5/lib/python3.6/site-packages/rest_framework/relations.py", line 542, in to_representation
Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: for value in iterable
Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File "/opt/python/3.6.5/lib/python3.6/site-packages/rest_framework/relations.py", line 542, in <listcomp>
Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: for value in iterable
Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File "/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore-3.0.0rc3.dev0-py3.6.egg/pulpcore/app/serializers/task.py", line 25, in to_representation
Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: return serializer.data.get('_href')
Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File "/opt/python/3.6.5/lib/python3.6/site-packages/rest_framework/serializers.py", line 563, in data
Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: ret = super(Serializer, self).data
Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File "/opt/python/3.6.5/lib/python3.6/site-packages/rest_framework/serializers.py", line 262, in data
Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: self._data = self.to_representation(self.instance)
Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File "/opt/python/3.6.5/lib/python3.6/site-packages/rest_framework/serializers.py", line 530, in to_representation
Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: ret[field.field_name] = field.to_representation(attribute)
Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File "/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore-3.0.0rc3.dev0-py3.6.egg/pulpcore/app/serializers/fields.py", line 248, in to_representation
Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: host.strip('/'),
Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: AttributeError: 'NoneType' object has no attribute 'strip'
Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: 127.0.0.1 - admin [25/Jun/2019:17:53:08 +0000] "GET /pulp/api/v3/tasks/ HTTP/1.1" 500 27 "-" "HTTPie/0.9.4
From: dkliban at redhat.com At: 06/25/19 13:46:18To: Bin Li (BLOOMBERG/ 120 PARK )
Cc: pulp-list at redhat.com
Subject: Re: [Pulp-list] Pulp 3 internal error
The logs are available in journalctl. I usually run 'sudo journalctl -l -f' to monitor the logs. What is the traceback that is emitted when you query this task?
A possibly related issue got filed today: https://pulp.plan.io/issues/4945
On Tue, Jun 25, 2019 at 1:35 PM Bin Li (BLOOMBERG/ 120 PARK) <bli111 at bloomberg.net> wrote:
I suddenly got a internal 500 error when query a task which creates a publication. Restarted all pulp daemons didn't help. Still able to GET repositories, publications, remotes information but query tasks and distributions gave me internal error. Database looks fine. I am able connect and login postgresql as pulp user. How do I troubleshot this issue?
used the following to create distribution:
http POST http://localhost:24817/pulp/api/v3/distributions/rpm/rpm/<<EOF
{
"base_path": "bloomberg/systems_third_party_rhel7.2",
"name": "systems_third_party_rhel7.2",
"publication": "/pulp/api/v3/publications/rpm/rpm/894a8109-001b-42fc-8658-4ac28bc52c0e/"
}
EOF
Below is current status:
{
"database_connection": {
"connected": true
},
"missing_workers": [],
"online_workers": [
{
"_created": "2019-06-20T18:34:01.015187Z",
"_href": "/pulp/api/v3/workers/fd7dc86c-3bff-4631-a86d-f49f618d3a79/",
"last_heartbeat": "2019-06-25T17:14:50.374396Z",
"missing": false,
"name": "resource-manager at ip-1-76-158-221.drr1.bcpc.bloomberg.com",
"online": true
},
{
"_created": "2019-06-20T18:34:01.347733Z",
"_href": "/pulp/api/v3/workers/b4dfbb8a-69f4-4411-9311-ff84a63756c0/",
"last_heartbeat": "2019-06-25T17:14:50.575492Z",
"missing": false,
"name": "reserved-resource-worker-1 at ip-1-76-158-221.drr1.bcpc.bloomberg.com",
"online": true
},
{
"_created": "2019-06-20T18:34:01.463024Z",
"_href": "/pulp/api/v3/workers/d1db2594-52b6-402e-8ef1-7c0a5635c3c4/",
"last_heartbeat": "2019-06-25T17:14:50.976399Z",
"missing": false,
"name": "reserved-resource-worker-2 at ip-1-76-158-221.drr1.bcpc.bloomberg.com",
"online": true
}
],
"redis_connection": {
"connected": true
},
"versions": [
{
"component": "pulpcore",
"version": "3.0.0rc3.dev0"
},
{
"component": "pulpcore-plugin",
"version": "0.1.0rc3.dev0"
},
{
"component": "pulp_rpm",
"version": "3.0.0b4.dev0"
}
]
}
_______________________________________________
Pulp-list mailing list
Pulp-list at redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list
_______________________________________________
Pulp-list mailing list
Pulp-list at redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list
_______________________________________________
Pulp-list mailing list
Pulp-list at redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20190625/0242b0aa/attachment.htm>
More information about the Pulp-list
mailing list