[Spacewalk-list] Inactive Systems

Mon Jun 5 19:31:49 UTC 2017

Did you rm the /etc/sysconfig/rhn/osad-auth.conf file and then restart jabberd?

--Matthew Wilkinson

From: spacewalk-list-bounces at redhat.com [mailto:spacewalk-list-bounces at redhat.com] On Behalf Of Daryl Rose
Sent: Monday, June 05, 2017 14:26
To: spacewalk-list at redhat.com
Subject: [Spacewalk-list] Inactive Systems

[This is an external email. Be cautious with links, attachments and responses.]
________________________________

Last week 218 systems became "Inactive Systems".  All of these systems are in the DMZ and connect via a proxy server.  I discovered that /var on the proxy had filled, and I'm guessing corrupted the Berkeley Database.  I shutdown Jabberd, cleaned everything out, restarted jabberd, and tried to re-authenticate the inactive systems.  However, I'm not able to get them to re-authenticate.

There are multiple errors in multiple error logs.

This is a snippet from the /var/log/httpd/error log:

[Mon Jun 05 10:41:19 2017] [error] Exception reported from <FQDN proxy server>
[Mon Jun 05 10:41:19 2017] [error] Time: Mon Jun  5 10:41:19 2017
[Mon Jun 05 10:41:19 2017] [error] Exception type <type 'exceptions.EOFError'>
[Mon Jun 05 10:41:19 2017] [error] Exception while handling function handler
[Mon Jun 05 10:41:19 2017] [error] Request object information:
[Mon Jun 05 10:41:19 2017] [error] URI: /XMLRPC
[Mon Jun 05 10:41:19 2017] [error] Remote Host: <web-server>
[Mon Jun 05 10:41:19 2017] [error] Server Name: <FQDN proxy server>:443
[Mon Jun 05 10:41:19 2017] [error] Headers passed in:
[Mon Jun 05 10:41:19 2017] [error] \tAccept-Encoding: identity
[Mon Jun 05 10:41:19 2017] [error] \tCONTENT_LENGTH: 122
[Mon Jun 05 10:41:19 2017] [error] \tCONTENT_TYPE: text/xml
[Mon Jun 05 10:41:19 2017] [error] \tDOCUMENT_ROOT: /var/www/html
[Mon Jun 05 10:41:19 2017] [error] \tGATEWAY_INTERFACE: CGI/1.1
[Mon Jun 05 10:41:19 2017] [error] \tHTTPS: 1
[Mon Jun 05 10:41:19 2017] [error] \tHTTP_ACCEPT_ENCODING: identity
[Mon Jun 05 10:41:19 2017] [error] \tHTTP_HOST: <FQDN proxy server>
[Mon Jun 05 10:41:19 2017] [error] \tHTTP_USER_AGENT: rhn.rpclib.py/2.5.75-1.el6
[Mon Jun 05 10:41:19 2017] [error] \tHTTP_X_CLIENT_VERSION: 1
[Mon Jun 05 10:41:19 2017] [error] \tHTTP_X_INFO: RPC Processor (C) Red Hat, Inc (version 2.5.75-1.el6)
[Mon Jun 05 10:41:19 2017] [error] \tHTTP_X_RHN_TRANSPORT_CAPABILITY: follow-redirects=3
[Mon Jun 05 10:41:19 2017] [error] \tHTTP_X_TRANSPORT_INFO: Extended Capabilities Transport (C) Red Hat, Inc (version 2.5.75-1.el6)
[Mon Jun 05 10:41:19 2017] [error] \tHost: <FQDN proxy server>
[Mon Jun 05 10:41:19 2017] [error] \tPATH_INFO:
[Mon Jun 05 10:41:19 2017] [error] \tQUERY_STRING:
[Mon Jun 05 10:41:19 2017] [error] \tREMOTE_ADDR: 172.19.216.14
[Mon Jun 05 10:41:19 2017] [error] \tREMOTE_PORT: 43218
[Mon Jun 05 10:41:19 2017] [error] \tREQUEST_METHOD: POST
[Mon Jun 05 10:41:19 2017] [error] \tREQUEST_URI: /XMLRPC
[Mon Jun 05 10:41:19 2017] [error] \tSCRIPT_FILENAME: /usr/share/rhn/wsgi/xmlrpc.py
[Mon Jun 05 10:41:19 2017] [error] \tSCRIPT_NAME: /XMLRPC
[Mon Jun 05 10:41:19 2017] [error] \tSCRIPT_URI: https://<FQDN<https://%3cFQDN> proxy server>/XMLRPC
[Mon Jun 05 10:41:19 2017] [error] \tSCRIPT_URL: /XMLRPC
[Mon Jun 05 10:41:19 2017] [error] \tSERVER_ADDR: 172.17.210.59
[Mon Jun 05 10:41:19 2017] [error] \tSERVER_ADMIN: root at localhost
[Mon Jun 05 10:41:19 2017] [error] \tSERVER_NAME: <FQDN proxy server>
[Mon Jun 05 10:41:19 2017] [error] \tSERVER_PORT: 443
[Mon Jun 05 10:41:19 2017] [error] \tSERVER_PROTOCOL: HTTP/1.1
[Mon Jun 05 10:41:19 2017] [error] \tSERVER_SIGNATURE: <address>Apache/2.2.15 (Red Hat) Server at <FQDN proxy server> Port 443</address>
[Mon Jun 05 10:41:19 2017] [error]
[Mon Jun 05 10:41:19 2017] [error] \tSERVER_SOFTWARE: Apache/2.2.15 (Red Hat)
[Mon Jun 05 10:41:19 2017] [error] \tUser-Agent: rhn.rpclib.py/2.5.75-1.el6
[Mon Jun 05 10:41:19 2017] [error] \tX-Client-Version: 1
[Mon Jun 05 10:41:19 2017] [error] \tX-Info: RPC Processor (C) Red Hat, Inc (version 2.5.75-1.el6)
[Mon Jun 05 10:41:19 2017] [error] \tX-RHN-Transport-Capability: follow-redirects=3
[Mon Jun 05 10:41:19 2017] [error] \tX-Transport-Info: Extended Capabilities Transport (C) Red Hat, Inc (version 2.5.75-1.el6)
[Mon Jun 05 10:41:19 2017] [error] \tmod_ssl.is_https: <built-in method ssl_is_https of mod_wsgi.Adapter object at 0x7fd673532a08>
[Mon Jun 05 10:41:19 2017] [error] \tmod_ssl.var_lookup: <built-in method ssl_var_lookup of mod_wsgi.Adapter object at 0x7fd673532a08>
[Mon Jun 05 10:41:19 2017] [error] \tmod_wsgi.application_group: corp-spwlkprxy-prod01.dtndmz.com|/xmlrpc
[Mon Jun 05 10:41:19 2017] [error] \tmod_wsgi.callable_object: application
[Mon Jun 05 10:41:19 2017] [error] \tmod_wsgi.handler_script:
[Mon Jun 05 10:41:19 2017] [error] \tmod_wsgi.input_chunked: 0
[Mon Jun 05 10:41:19 2017] [error] \tmod_wsgi.listener_host:
[Mon Jun 05 10:41:19 2017] [error] \tmod_wsgi.listener_port: 443
[Mon Jun 05 10:41:19 2017] [error] \tmod_wsgi.process_group:
[Mon Jun 05 10:41:19 2017] [error] \tmod_wsgi.request_handler: wsgi-script
[Mon Jun 05 10:41:19 2017] [error] \tmod_wsgi.script_reloading: 1
[Mon Jun 05 10:41:19 2017] [error] \tmod_wsgi.version: (3, 2)
[Mon Jun 05 10:41:19 2017] [error] \twsgi.errors: <mod_wsgi.Log object at 0x7fd673415ab0>
[Mon Jun 05 10:41:19 2017] [error] \twsgi.file_wrapper: <built-in method file_wrapper of mod_wsgi.Adapter object at 0x7fd673532a08>
[Mon Jun 05 10:41:19 2017] [error] \twsgi.input: <cStringIO.StringO object at 0x7fd6733ffab0>
[Mon Jun 05 10:41:19 2017] [error] \twsgi.multiprocess: True
[Mon Jun 05 10:41:19 2017] [error] \twsgi.multithread: False
[Mon Jun 05 10:41:19 2017] [error] \twsgi.run_once: False
[Mon Jun 05 10:41:19 2017] [error] \twsgi.url_scheme: https
[Mon Jun 05 10:41:19 2017] [error] \twsgi.version: (1, 1)
[Mon Jun 05 10:41:19 2017] [error] Extra information about this error:
[Mon Jun 05 10:41:19 2017] [error] Unhandled exception type
[Mon Jun 05 10:41:19 2017] [error]
[Mon Jun 05 10:41:19 2017] [error] Exception Handler Information
[Mon Jun 05 10:41:19 2017] [error] Traceback (most recent call last):
[Mon Jun 05 10:41:19 2017] [error]   File "/usr/share/rhn/proxy/apacheServer.py", line 63, in __call__
[Mon Jun 05 10:41:19 2017] [error]     ret = f(req)
[Mon Jun 05 10:41:19 2017] [error]   File "/usr/share/rhn/proxy/apacheHandler.py", line 357, in handler
[Mon Jun 05 10:41:19 2017] [error]     ret = handlerObj.handler()
[Mon Jun 05 10:41:19 2017] [error]   File "/usr/share/rhn/proxy/broker/rhnBroker.py", line 220, in handler
[Mon Jun 05 10:41:19 2017] [error]     authToken = self.proxyAuth.check_cached_token()
[Mon Jun 05 10:41:19 2017] [error]   File "/usr/share/rhn/proxy/rhnProxyAuth.py", line 130, in check_cached_token
[Mon Jun 05 10:41:19 2017] [error]     oldToken = self.get_cached_token()
[Mon Jun 05 10:41:19 2017] [error]   File "/usr/share/rhn/proxy/rhnProxyAuth.py", line 147, in get_cached_token
[Mon Jun 05 10:41:19 2017] [error]     return shelf[key]
[Mon Jun 05 10:41:19 2017] [error]   File "/usr/share/rhn/proxy/rhnProxyAuth.py", line 450, in __getitem__
[Mon Jun 05 10:41:19 2017] [error]     val = rhnCache.get(rkey, missing_is_null=0)
[Mon Jun 05 10:41:19 2017] [error]   File "/usr/lib/python2.6/site-packages/spacewalk/common/rhnCache.py", line 70, in get
[Mon Jun 05 10:41:19 2017] [error]     return cache.get(name, modified)
[Mon Jun 05 10:41:19 2017] [error]   File "/usr/lib/python2.6/site-packages/spacewalk/common/rhnCache.py", line 366, in get
[Mon Jun 05 10:41:19 2017] [error]     return cPickle.loads(pickled)
[Mon Jun 05 10:41:19 2017] [error] EOFError

This is from one of the clients that can't connect:

2017-06-05 12:46:48 rhn_log.log_error: Error caught:
2017-06-05 12:46:48 rhn_log.log_error: Traceback (most recent call last):
  File "/usr/share/rhn/osad/jabber_lib.py", line 127, in main
    self.fix_connection(c)
  File "/usr/share/rhn/osad/osad.py", line 202, in fix_connection
    server.registration.welcome_message()
  File "/usr/lib64/python2.6/site-packages/rhn/rpclib.py", line 662, in __call__
    return self._send(self._name, args)
  File "/usr/lib64/python2.6/site-packages/rhn/rpclib.py", line 394, in _request
    self._handler, request, verbose=self._verbose)
  File "/usr/lib64/python2.6/site-packages/rhn/transports.py", line 177, in request
    headers, fd = req.send_http(host, handler)
  File "/usr/lib64/python2.6/site-packages/rhn/transports.py", line 740, in send_http
    response.status, response.reason, response.msg)
ProtocolError: <ProtocolError for <FQDN Proxy Server> /XMLRPC: 500 Internal Server Error>

My inital thoughts turn to the certifiacation, but It that shouldn't be the case, because this has been working fine for over a year.  As a matter of fact, I can see that the client estabilishes a connection on with the proxy server on port 5222.

Anyone have any thoughts on what could be causing this?

Thank you.

Daryl

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/spacewalk-list/attachments/20170605/3f4cf204/attachment.htm>