[Spacewalk-list] Inactive Systems

Tue Jun 6 18:22:49 UTC 2017

I think that I figured out what is wrong, I just need to figure out how to resolve it.

Last November, I upgraded the SW server to v2.6, and a few months ago, I changed jabberd to use the Postgres DB.  However, the proxy server is still on v2.4 (I forgot all about upgrading it when I did the primary server).

I started reviewing the jabberd config files, c2s.xml, s2s.xml and sm.xml and noticed that the current config files are significatly differnet from the dist files, plus they had the changes that I made when switching to the Postgres DB.  My guess is that SW must have updated the config files when I made the changes a few months ago.

I copied the .dist files back to being the primary config file and changed the ID and password everywhere that I found the settings, but I'm still not able to connect.

Besides the server ID and DB password, is there anything else in the config file that I need to change? Do I need to upgrade to v2.6 and install Postgres on the proxy to make everything work?

Thanks

Daryl

________________________________
From: spacewalk-list-bounces at redhat.com <spacewalk-list-bounces at redhat.com> on behalf of Daryl Rose <darylrose at outlook.com>
Sent: Tuesday, June 6, 2017 10:10 AM
To: spacewalk-list at redhat.com
Subject: Re: [Spacewalk-list] Inactive Systems

Yes I did and I get:

service osad start
Starting osad: 2017-06-06 15:08:27 jabber_lib.main: Unable to connect to jabber servers, sleeping 101 seconds

Any thoughts?

Thank you.

Daryl

________________________________
From: spacewalk-list-bounces at redhat.com <spacewalk-list-bounces at redhat.com> on behalf of Wilkinson, Matthew <MatthewWilkinson at alliantenergy.com>
Sent: Monday, June 5, 2017 2:31 PM
To: spacewalk-list at redhat.com
Subject: Re: [Spacewalk-list] Inactive Systems

Did you rm the /etc/sysconfig/rhn/osad-auth.conf file and then restart jabberd?

--Matthew Wilkinson

From: spacewalk-list-bounces at redhat.com [mailto:spacewalk-list-bounces at redhat.com] On Behalf Of Daryl Rose
Sent: Monday, June 05, 2017 14:26
To: spacewalk-list at redhat.com
Subject: [Spacewalk-list] Inactive Systems

[This is an external email. Be cautious with links, attachments and responses.]

________________________________

Last week 218 systems became "Inactive Systems".  All of these systems are in the DMZ and connect via a proxy server.  I discovered that /var on the proxy had filled, and I'm guessing corrupted the Berkeley Database.  I shutdown Jabberd, cleaned everything out, restarted jabberd, and tried to re-authenticate the inactive systems.  However, I'm not able to get them to re-authenticate.

There are multiple errors in multiple error logs.

This is a snippet from the /var/log/httpd/error log:

[Mon Jun 05 10:41:19 2017] [error] Exception reported from <FQDN proxy server>

[Mon Jun 05 10:41:19 2017] [error] Time: Mon Jun  5 10:41:19 2017

[Mon Jun 05 10:41:19 2017] [error] Exception type <type 'exceptions.EOFError'>

[Mon Jun 05 10:41:19 2017] [error] Exception while handling function handler

[Mon Jun 05 10:41:19 2017] [error] Request object information:

[Mon Jun 05 10:41:19 2017] [error] URI: /XMLRPC

[Mon Jun 05 10:41:19 2017] [error] Remote Host: <web-server>

[Mon Jun 05 10:41:19 2017] [error] Server Name: <FQDN proxy server>:443

[Mon Jun 05 10:41:19 2017] [error] Headers passed in:

[Mon Jun 05 10:41:19 2017] [error] \tAccept-Encoding: identity

[Mon Jun 05 10:41:19 2017] [error] \tCONTENT_LENGTH: 122

[Mon Jun 05 10:41:19 2017] [error] \tCONTENT_TYPE: text/xml

[Mon Jun 05 10:41:19 2017] [error] \tDOCUMENT_ROOT: /var/www/html

[Mon Jun 05 10:41:19 2017] [error] \tGATEWAY_INTERFACE: CGI/1.1

[Mon Jun 05 10:41:19 2017] [error] \tHTTPS: 1

[Mon Jun 05 10:41:19 2017] [error] \tHTTP_ACCEPT_ENCODING: identity

[Mon Jun 05 10:41:19 2017] [error] \tHTTP_HOST: <FQDN proxy server>

[Mon Jun 05 10:41:19 2017] [error] \tHTTP_USER_AGENT: rhn.rpclib.py/2.5.75-1.el6

[Mon Jun 05 10:41:19 2017] [error] \tHTTP_X_CLIENT_VERSION: 1

[Mon Jun 05 10:41:19 2017] [error] \tHTTP_X_INFO: RPC Processor (C) Red Hat, Inc (version 2.5.75-1.el6)

[Mon Jun 05 10:41:19 2017] [error] \tHTTP_X_RHN_TRANSPORT_CAPABILITY: follow-redirects=3

[Mon Jun 05 10:41:19 2017] [error] \tHTTP_X_TRANSPORT_INFO: Extended Capabilities Transport (C) Red Hat, Inc (version 2.5.75-1.el6)

[Mon Jun 05 10:41:19 2017] [error] \tHost: <FQDN proxy server>

[Mon Jun 05 10:41:19 2017] [error] \tPATH_INFO:

[Mon Jun 05 10:41:19 2017] [error] \tQUERY_STRING:

[Mon Jun 05 10:41:19 2017] [error] \tREMOTE_ADDR: 172.19.216.14

[Mon Jun 05 10:41:19 2017] [error] \tREMOTE_PORT: 43218

[Mon Jun 05 10:41:19 2017] [error] \tREQUEST_METHOD: POST

[Mon Jun 05 10:41:19 2017] [error] \tREQUEST_URI: /XMLRPC

[Mon Jun 05 10:41:19 2017] [error] \tSCRIPT_FILENAME: /usr/share/rhn/wsgi/xmlrpc.py

[Mon Jun 05 10:41:19 2017] [error] \tSCRIPT_NAME: /XMLRPC

[Mon Jun 05 10:41:19 2017] [error] \tSCRIPT_URI: https://<FQDN<https://%3cFQDN> proxy server>/XMLRPC

[Mon Jun 05 10:41:19 2017] [error] \tSCRIPT_URL: /XMLRPC

[Mon Jun 05 10:41:19 2017] [error] \tSERVER_ADDR: 172.17.210.59

[Mon Jun 05 10:41:19 2017] [error] \tSERVER_ADMIN: root at localhost

[Mon Jun 05 10:41:19 2017] [error] \tSERVER_NAME: <FQDN proxy server>

[Mon Jun 05 10:41:19 2017] [error] \tSERVER_PORT: 443

[Mon Jun 05 10:41:19 2017] [error] \tSERVER_PROTOCOL: HTTP/1.1

[Mon Jun 05 10:41:19 2017] [error] \tSERVER_SIGNATURE: <address>Apache/2.2.15 (Red Hat) Server at <FQDN proxy server> Port 443</address>

[Mon Jun 05 10:41:19 2017] [error]

[Mon Jun 05 10:41:19 2017] [error] \tSERVER_SOFTWARE: Apache/2.2.15 (Red Hat)

[Mon Jun 05 10:41:19 2017] [error] \tUser-Agent: rhn.rpclib.py/2.5.75-1.el6

[Mon Jun 05 10:41:19 2017] [error] \tX-Client-Version: 1

[Mon Jun 05 10:41:19 2017] [error] \tX-Info: RPC Processor (C) Red Hat, Inc (version 2.5.75-1.el6)

[Mon Jun 05 10:41:19 2017] [error] \tX-RHN-Transport-Capability: follow-redirects=3

[Mon Jun 05 10:41:19 2017] [error] \tX-Transport-Info: Extended Capabilities Transport (C) Red Hat, Inc (version 2.5.75-1.el6)

[Mon Jun 05 10:41:19 2017] [error] \tmod_ssl.is_https: <built-in method ssl_is_https of mod_wsgi.Adapter object at 0x7fd673532a08>

[Mon Jun 05 10:41:19 2017] [error] \tmod_ssl.var_lookup: <built-in method ssl_var_lookup of mod_wsgi.Adapter object at 0x7fd673532a08>

[Mon Jun 05 10:41:19 2017] [error] \tmod_wsgi.application_group: corp-spwlkprxy-prod01.dtndmz.com|/xmlrpc

[Mon Jun 05 10:41:19 2017] [error] \tmod_wsgi.callable_object: application

[Mon Jun 05 10:41:19 2017] [error] \tmod_wsgi.handler_script:

[Mon Jun 05 10:41:19 2017] [error] \tmod_wsgi.input_chunked: 0

[Mon Jun 05 10:41:19 2017] [error] \tmod_wsgi.listener_host:

[Mon Jun 05 10:41:19 2017] [error] \tmod_wsgi.listener_port: 443

[Mon Jun 05 10:41:19 2017] [error] \tmod_wsgi.process_group:

[Mon Jun 05 10:41:19 2017] [error] \tmod_wsgi.request_handler: wsgi-script

[Mon Jun 05 10:41:19 2017] [error] \tmod_wsgi.script_reloading: 1

[Mon Jun 05 10:41:19 2017] [error] \tmod_wsgi.version: (3, 2)

[Mon Jun 05 10:41:19 2017] [error] \twsgi.errors: <mod_wsgi.Log object at 0x7fd673415ab0>

[Mon Jun 05 10:41:19 2017] [error] \twsgi.file_wrapper: <built-in method file_wrapper of mod_wsgi.Adapter object at 0x7fd673532a08>

[Mon Jun 05 10:41:19 2017] [error] \twsgi.input: <cStringIO.StringO object at 0x7fd6733ffab0>

[Mon Jun 05 10:41:19 2017] [error] \twsgi.multiprocess: True

[Mon Jun 05 10:41:19 2017] [error] \twsgi.multithread: False

[Mon Jun 05 10:41:19 2017] [error] \twsgi.run_once: False

[Mon Jun 05 10:41:19 2017] [error] \twsgi.url_scheme: https

[Mon Jun 05 10:41:19 2017] [error] \twsgi.version: (1, 1)

[Mon Jun 05 10:41:19 2017] [error] Extra information about this error:

[Mon Jun 05 10:41:19 2017] [error] Unhandled exception type

[Mon Jun 05 10:41:19 2017] [error]

[Mon Jun 05 10:41:19 2017] [error] Exception Handler Information

[Mon Jun 05 10:41:19 2017] [error] Traceback (most recent call last):

[Mon Jun 05 10:41:19 2017] [error]   File "/usr/share/rhn/proxy/apacheServer.py", line 63, in __call__

[Mon Jun 05 10:41:19 2017] [error]     ret = f(req)

[Mon Jun 05 10:41:19 2017] [error]   File "/usr/share/rhn/proxy/apacheHandler.py", line 357, in handler

[Mon Jun 05 10:41:19 2017] [error]     ret = handlerObj.handler()

[Mon Jun 05 10:41:19 2017] [error]   File "/usr/share/rhn/proxy/broker/rhnBroker.py", line 220, in handler

[Mon Jun 05 10:41:19 2017] [error]     authToken = self.proxyAuth.check_cached_token()

[Mon Jun 05 10:41:19 2017] [error]   File "/usr/share/rhn/proxy/rhnProxyAuth.py", line 130, in check_cached_token

[Mon Jun 05 10:41:19 2017] [error]     oldToken = self.get_cached_token()

[Mon Jun 05 10:41:19 2017] [error]   File "/usr/share/rhn/proxy/rhnProxyAuth.py", line 147, in get_cached_token

[Mon Jun 05 10:41:19 2017] [error]     return shelf[key]

[Mon Jun 05 10:41:19 2017] [error]   File "/usr/share/rhn/proxy/rhnProxyAuth.py", line 450, in __getitem__

[Mon Jun 05 10:41:19 2017] [error]     val = rhnCache.get(rkey, missing_is_null=0)

[Mon Jun 05 10:41:19 2017] [error]   File "/usr/lib/python2.6/site-packages/spacewalk/common/rhnCache.py", line 70, in get

[Mon Jun 05 10:41:19 2017] [error]     return cache.get(name, modified)

[Mon Jun 05 10:41:19 2017] [error]   File "/usr/lib/python2.6/site-packages/spacewalk/common/rhnCache.py", line 366, in get

[Mon Jun 05 10:41:19 2017] [error]     return cPickle.loads(pickled)

[Mon Jun 05 10:41:19 2017] [error] EOFError

This is from one of the clients that can't connect:

2017-06-05 12:46:48 rhn_log.log_error: Error caught:

2017-06-05 12:46:48 rhn_log.log_error: Traceback (most recent call last):

  File "/usr/share/rhn/osad/jabber_lib.py", line 127, in main

    self.fix_connection(c)

  File "/usr/share/rhn/osad/osad.py", line 202, in fix_connection

    server.registration.welcome_message()

  File "/usr/lib64/python2.6/site-packages/rhn/rpclib.py", line 662, in __call__

    return self._send(self._name, args)

  File "/usr/lib64/python2.6/site-packages/rhn/rpclib.py", line 394, in _request

    self._handler, request, verbose=self._verbose)

  File "/usr/lib64/python2.6/site-packages/rhn/transports.py", line 177, in request

    headers, fd = req.send_http(host, handler)

  File "/usr/lib64/python2.6/site-packages/rhn/transports.py", line 740, in send_http

    response.status, response.reason, response.msg)

ProtocolError: <ProtocolError for <FQDN Proxy Server> /XMLRPC: 500 Internal Server Error>

My inital thoughts turn to the certifiacation, but It that shouldn't be the case, because this has been working fine for over a year.  As a matter of fact, I can see that the client estabilishes a connection on with the proxy server on port 5222.

Anyone have any thoughts on what could be causing this?

Thank you.

Daryl

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/spacewalk-list/attachments/20170606/0894a12a/attachment.htm>