[Spacewalk-list] osad fails to pick up queued actions, when client is behind proxy server...

Tue Mar 11 14:47:01 UTC 2014

Thanks for the reply; the hostnames are in DNS and nslookup responds... I have some questions about that, since we're using load-balancers to route to a virtual-ip address, to accommodate a switch-over from the primary server to standby for patching and maintenance activities...

Reverse lookup (ip->hostname) doesn't resolve to the hostname expected.
Nslookup returns the load-balancer's vip hostname as an alias to another hostname which includes an additional field in its hostname... the name resolution seems to work fine though.

I'm curious about the reverse lookup however. I am able to get the client->proxy->Spacewalk osad connections to work with the original hostnames (not the Load-balancer VIP hostname alias)... those hostnames are fully resolvable forward and reverse... and they appear to work. My next step today is to resolve the reverse lookup.

I've loaded ssldump, and I'm just now trying to get samples from eth0, but I'm getting  some "Length mismatch" errors while running it... If I get more detail, I'll post it.

Best Regards,

Whit

--
Whitney "Whit" Latta
Sr. Systems Engineer
Linux - Open Systems Engineering
whitney_latta at homedepot.com<mailto:whitney_latta at homedepot.com>
Office: (770) 433-8211 ext:81278
Mobile: (678) 231-2049

--
"We've done the impossible, and that makes us mighty." ―Malcolm Reynolds
--

From: spacewalk-list-bounces at redhat.com [mailto:spacewalk-list-bounces at redhat.com] On Behalf Of Daniel Kupfer
Sent: Tuesday, March 11, 2014 10:26 AM
To: spacewalk-list at redhat.com
Subject: Re: [Spacewalk-list] osad fails to pick up queued actions, when client is behind proxy server...

Whitney, i'm new to this stuff but i had problems with the jabberd too. Are you're hostnames all in DNS and lookup-able?

Regards,
Daniel Kupfer.

On Tue, Mar 11, 2014 at 2:51 PM, <WHITNEY_LATTA at homedepot.com<mailto:WHITNEY_LATTA at homedepot.com>> wrote:
Hello,

I've pretty much run out of ideas to troubleshoot this issue further; I am hoping someone can suggest some more in-depth troubleshooting of the jabberd communications in Spacewalk v2.0. Googling the symptoms, I found several others with the same issue, but no clear resolution... everything that has previously been suggested has come up empty.

The issue we're seeing is that osad and osa-dispatcher on the client and server side appear to connect to the proxy-server, but osa ping from the Spacewalk gui does nothing, and any actions queued up to the client are not picked up until the next rhn_check is exec'ed manually or from rhnsd.  Running rhn_check works fine; in fact everything else in Spacewalk works fine, except the osa-dispatcher push to the clients.

So far, we've tried shutting down all the jabberd and osa services on the Spacewalk, proxy, and client servers; then, clear the /var/lib/jabberd/db/* files on the Spacewalk and proxy servers; then, remove the osad-auth.conf file on the proxy and client... restart everything in order (Spacewalk, proxy, client)... the issue persists... ping to the proxy server returns fine, but not for the end client. If the client is set to connect to the Spacewalk server, it works, but not through the proxy server.

Checked the ssl certs and checksums/permissions appear correct. The OU and CN point to the spacewalk server hostname. I even tried spacewalk-hostname-rename to no avail.

The one gotcha in this is that we're going through a citrix netscaler load-balancer in front of the Spacewalk server (actually a pair, in an active-standby configuration, so only 1 is up at any one time). If I eliminate the LB from the configuration and setup with the original FQDN hostnames, the jabberd communications through osad appears to work fine from the client through the proxy server to Spacewalk. It's just frustrating that the connections appear to be made, but no communication is going through in this configuration, and no errors or other messages can be found to identify where it is failing... It appears to be ssl that is failing, but how to isolate the root cause?

Is there some way to generate more debug output specifically from the connection, and in particular the ssl interactions? I set debug levels in osad to 7 and in rhn.conf to 5, and it generates a lot of output, but I can't get any details on what is happening between the client, through the proxy server to Spacewalk... or I'm missing something.

Curiously, I do see that osa-dispatcher doesn't seem to find any servers to ping:

osad/osa_dispatcher.process_once('Clients to be pinged:', None)

Looking at the rhnpushclient table, it appears the ping was entered, but the LAST_PING_TIME field is never cleared (until I restart the services):

SQL> select ID,JABBER_ID,LAST_PING_TIME from rhnpushclient;

        ID
----------
JABBER_ID
--------------------------------------------------------------------------------
LAST_PING_TIME
---------------------------------------------------------------------------
        11
osad-905526b856@<proxy-server-FQDN>/osad
11-MAR-14 09.38.56.615000 AM

         9
osad-85d7c101c5@<spacewalk-server-FQDN>/osad

Any help would be greatly appreciated.

Best Regards,

Whit

--
Whitney "Whit" Latta
Sr. Systems Engineer
Linux - Open Systems Engineering

--
"We've done the impossible, and that makes us mighty." ―Malcolm Reynolds
--

________________________________

The information in this Internet Email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this Email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed to our clients any opinions or advice contained in this Email are subject to the terms and conditions expressed in any applicable governing The Home Depot terms of business or client engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy and content of this attachment and for any damages or losses arising from any inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature, which may be contained in this attachment and shall not be liable for direct, indirect, consequential or special damages in connection with this e-mail message or its attachment.

_______________________________________________
Spacewalk-list mailing list
Spacewalk-list at redhat.com<mailto:Spacewalk-list at redhat.com>
https://www.redhat.com/mailman/listinfo/spacewalk-list

________________________________

The information in this Internet Email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this Email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed to our clients any opinions or advice contained in this Email are subject to the terms and conditions expressed in any applicable governing The Home Depot terms of business or client engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy and content of this attachment and for any damages or losses arising from any inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature, which may be contained in this attachment and shall not be liable for direct, indirect, consequential or special damages in connection with this e-mail message or its attachment.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/spacewalk-list/attachments/20140311/3f0cb387/attachment.htm>