[Spacewalk-list] 1 proxy cant download channel's /repodata/primary.xml.gz

Fri Feb 20 16:37:15 UTC 2015

I have a spacewalk installation with a mater and a few proxies.   All of a sudden today, one of my more remote proxies cannot serve repodata/primary.xml.gz for the channel.  It really looks like nothing changed on that proxy and its been in service for a couple months now.

When clients hit the proxy, it serves a 404 for that file.

The error log for a request:
[Fri Feb 20 08:23:40 2015] [error] Exception reported from <hostname scrubbed>
[Fri Feb 20 08:23:40 2015] [error] Time: Fri Feb 20 08:23:40 2015
[Fri Feb 20 08:23:40 2015] [error] Exception type <class 'socket.timeout'>
[Fri Feb 20 08:23:40 2015] [error] Exception while handling function SharedHandler._serverCommo
[Fri Feb 20 08:23:40 2015] [error] Request object information:
[Fri Feb 20 08:23:40 2015] [error] URI: /XMLRPC/GET-REQ/<channel name scrubbed> /repodata/primary.xml.gz
[Fri Feb 20 08:23:40 2015] [error] Remote Host: <hostname scrubbed>
[Fri Feb 20 08:23:40 2015] [error] Server Name: <hostname scrubbed>:80
[Fri Feb 20 08:23:40 2015] [error] Headers passed in:
[Fri Feb 20 08:23:40 2015] [error] \tAccept: */*
[Fri Feb 20 08:23:40 2015] [error] \tDOCUMENT_ROOT: /var/www/html
[Fri Feb 20 08:23:40 2015] [error] \tGATEWAY_INTERFACE: CGI/1.1
[Fri Feb 20 08:23:40 2015] [error] \tHTTP_ACCEPT: */*
[Fri Feb 20 08:23:40 2015] [error] \tHTTP_HOST: <hostname scrubbed>
[Fri Feb 20 08:23:40 2015] [error] \tHTTP_PRAGMA: no-cache
[Fri Feb 20 08:23:40 2015] [error] \tHTTP_USER_AGENT: rhn.rpclib.py/2.5.72-1.el6
[Fri Feb 20 08:23:40 2015] [error] \tHTTP_X_LIBCURL_EMPTY_HEADER_WORKAROUND: *
[Fri Feb 20 08:23:40 2015] [error] \tHTTP_X_RHN_AUTH: apQHxvpocp9VUQKDAKU9whrNlbnhbimIwxV+f+qSaLA=
[Fri Feb 20 08:23:40 2015] [error] \tHTTP_X_RHN_AUTH_EXPIRE_OFFSET: 3600.0
[Fri Feb 20 08:23:40 2015] [error] \tHTTP_X_RHN_AUTH_SERVER_TIME: 1424449299.62
[Fri Feb 20 08:23:40 2015] [error] \tHTTP_X_RHN_AUTH_USER_ID:
[Fri Feb 20 08:23:40 2015] [error] \tHTTP_X_RHN_SERVER_ID: 1000010032
[Fri Feb 20 08:23:40 2015] [error] \tHTTP_X_RHN_TRANSPORT_CAPABILITY: follow-redirects=3
[Fri Feb 20 08:23:40 2015] [error] \tHost: <hostname scrubbed>
[Fri Feb 20 08:23:40 2015] [error] \tPATH_INFO: /GET-REQ/<channel name scrubbed>/repodata/primary.xml.gz
[Fri Feb 20 08:23:40 2015] [error] \tPATH_TRANSLATED: /var/www/html/GET-REQ/<channel name scrubbed>/repodata/primary.xml.gz
[Fri Feb 20 08:23:40 2015] [error] \tPragma: no-cache
[Fri Feb 20 08:23:40 2015] [error] \tQUERY_STRING:
[Fri Feb 20 08:23:40 2015] [error] \tREMOTE_ADDR: <IP scrubbed>
[Fri Feb 20 08:23:40 2015] [error] \tREMOTE_PORT: 60139
[Fri Feb 20 08:23:40 2015] [error] \tREQUEST_METHOD: GET
[Fri Feb 20 08:23:40 2015] [error] \tREQUEST_URI: /XMLRPC/GET-REQ/<channel name scrubbed>/repodata/primary.xml.gz
[Fri Feb 20 08:23:40 2015] [error] \tSCRIPT_FILENAME: /usr/share/rhn/wsgi/xmlrpc.py
[Fri Feb 20 08:23:40 2015] [error] \tSCRIPT_NAME: /XMLRPC
[Fri Feb 20 08:23:40 2015] [error] \tSCRIPT_URI: http://<hostname<http://%3chostname> scrubbed>/XMLRPC/GET-REQ/<channel name scrubbed>/repodata/primary.xml.gz
[Fri Feb 20 08:23:40 2015] [error] \tSCRIPT_URL: /XMLRPC/GET-REQ/<channel  name scrubbed>/repodata/primary.xml.gz
[Fri Feb 20 08:23:40 2015] [error] \tSERVER_ADDR: 172.23.1.93
[Fri Feb 20 08:23:40 2015] [error] \tSERVER_ADMIN: root at localhost
[Fri Feb 20 08:23:40 2015] [error] \tSERVER_NAME: <hostname scrubbed>
[Fri Feb 20 08:23:40 2015] [error] \tSERVER_PORT: 80
[Fri Feb 20 08:23:40 2015] [error] \tSERVER_PROTOCOL: HTTP/1.1
[Fri Feb 20 08:23:40 2015] [error] \tSERVER_SIGNATURE: <address>Apache/2.2.15 (CentOS) Server at <hostname scrubbed> Port 80</address>
[Fri Feb 20 08:23:40 2015] [error] \tSERVER_SOFTWARE: Apache/2.2.15 (CentOS)
[Fri Feb 20 08:23:40 2015] [error] \tUser-Agent: rhn.rpclib.py/2.5.72-1.el6
[Fri Feb 20 08:23:40 2015] [error] \tX-Libcurl-Empty-Header-Workaround: *
[Fri Feb 20 08:23:40 2015] [error] \tX-RHN-Auth: apQHxvpocp9VUQKDAKU9whrNlbnhbimIwxV+f+qSaLA=
[Fri Feb 20 08:23:40 2015] [error] \tX-RHN-Auth-Expire-Offset: 3600.0
[Fri Feb 20 08:23:40 2015] [error] \tX-RHN-Auth-Server-Time: 1424449299.62
[Fri Feb 20 08:23:40 2015] [error] \tX-RHN-Auth-User-Id:
[Fri Feb 20 08:23:40 2015] [error] \tX-RHN-Server-Id: 1000010032
[Fri Feb 20 08:23:40 2015] [error] \tX-RHN-Transport-Capability: follow-redirects=3
[Fri Feb 20 08:23:40 2015] [error] \tmod_ssl.is_https: <built-in method ssl_is_https of mod_wsgi.Adapter object at 0x7f29cd864030>
[Fri Feb 20 08:23:40 2015] [error] \tmod_ssl.var_lookup: <built-in method ssl_var_lookup of mod_wsgi.Adapter object at 0x7f29cd864030>
[Fri Feb 20 08:23:40 2015] [error] \tmod_wsgi.application_group: <hostname scrubbed>|/xmlrpc
[Fri Feb 20 08:23:40 2015] [error] \tmod_wsgi.callable_object: application
[Fri Feb 20 08:23:40 2015] [error] \tmod_wsgi.handler_script:
[Fri Feb 20 08:23:40 2015] [error] \tmod_wsgi.input_chunked: 0
[Fri Feb 20 08:23:40 2015] [error] \tmod_wsgi.listener_host:
[Fri Feb 20 08:23:40 2015] [error] \tmod_wsgi.listener_port: 80
[Fri Feb 20 08:23:40 2015] [error] \tmod_wsgi.process_group:
[Fri Feb 20 08:23:40 2015] [error] \tmod_wsgi.request_handler: wsgi-script
[Fri Feb 20 08:23:40 2015] [error] \tmod_wsgi.script_reloading: 1
[Fri Feb 20 08:23:40 2015] [error] \tmod_wsgi.version: (3, 2)
[Fri Feb 20 08:23:40 2015] [error] \twsgi.errors: <mod_wsgi.Log object at 0x7f29cd812f30>
[Fri Feb 20 08:23:40 2015] [error] \twsgi.file_wrapper: <built-in method file_wrapper of mod_wsgi.Adapter object at 0x7f29cd864030>
[Fri Feb 20 08:23:40 2015] [error] \twsgi.input: <mod_wsgi.Input object at 0x7f29cd812f70>
[Fri Feb 20 08:23:40 2015] [error] \twsgi.multiprocess: True
[Fri Feb 20 08:23:40 2015] [error] \twsgi.multithread: False
[Fri Feb 20 08:23:40 2015] [error] \twsgi.run_once: False
[Fri Feb 20 08:23:40 2015] [error] \twsgi.url_scheme: http
[Fri Feb 20 08:23:40 2015] [error] \twsgi.version: (1, 1)
[Fri Feb 20 08:23:40 2015] [error]
[Fri Feb 20 08:23:40 2015] [error] Exception Handler Information
[Fri Feb 20 08:23:40 2015] [error] Traceback (most recent call last):
[Fri Feb 20 08:23:40 2015] [error]   File "/usr/share/rhn/proxy/rhnShared.py", line 201, in _serverCommo
[Fri Feb 20 08:23:40 2015] [error]     status, headers, bodyFd = self._proxy2server()
[Fri Feb 20 08:23:40 2015] [error]   File "/usr/share/rhn/proxy/rhnShared.py", line 375, in _proxy2server
[Fri Feb 20 08:23:40 2015] [error]     response = http_connection.getresponse()
[Fri Feb 20 08:23:40 2015] [error]   File "/usr/lib/python2.6/site-packages/rhn/connections.py", line 93, in getresponse
[Fri Feb 20 08:23:40 2015] [error]     response.begin()
[Fri Feb 20 08:23:40 2015] [error]   File "/usr/lib64/python2.6/httplib.py", line 391, in begin
[Fri Feb 20 08:23:40 2015] [error]     version, status, reason = self._read_status()
[Fri Feb 20 08:23:40 2015] [error]   File "/usr/lib64/python2.6/httplib.py", line 349, in _read_status
[Fri Feb 20 08:23:40 2015] [error]     line = self.fp.readline()
[Fri Feb 20 08:23:40 2015] [error]   File "/usr/lib64/python2.6/socket.py", line 433, in readline
[Fri Feb 20 08:23:40 2015] [error]     data = recv(1)
[Fri Feb 20 08:23:40 2015] [error] timeout: timed out

The proxy is not having any actual network issues connecting to the master, and I can see in the UI that hosts going through that proxy are checking in as expected (one in the last minute even, so the networking is working fine, but it is a little slow since its remote).

All of my other proxies are not having this problem
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/spacewalk-list/attachments/20150220/9bf5d5da/attachment.htm>