[Spacewalk-list] Clients not picking up requested changes, period

Chris Swingler chris at chrisswingler.com
Tue Jan 27 16:48:13 UTC 2015 

More interesting things:

Using the reactivation feature doesn't work.

[root at zbx-prx-elk-04 log]# /usr/sbin/rhnreg_ks --force --serverUrl=
http://spacewalk.companyname.com/XMLRPC
--activationkey=9b601306ac1616daf1510d93a5084bf6
Error communicating with server. The message was:
Internal Server Error

Give it the wrong token, and you get an actual error:

[root at zbx-prx-elk-04 log]# /usr/sbin/rhnreg_ks --force --serverUrl=
http://spacewalk.companyname.com/XMLRPC
--activationkey=1-9b601306ac1616daf1510d93a5084bf6

Error Message:
    Could not find token '1-9b601306ac1616daf1510d93a5084bf6'
Error Class Code: 60
Error Class Info:
     The activation token specified could not be found on the server.
     Please retry with a valid key.



On Tue, Jan 27, 2015 at 10:36 AM, Chris Swingler <chris at chrisswingler.com>
wrote:

> Yeah, osad/rhnsd don't seem to have any bearing on this problem, turning
> them on/off and trying again was one of the first things I tried. No
> change; so it's not like osad is somehow half-picking up the task and then
> not finishing it.
>
> Worth mentioning, if it's not clear: Spacewalk never sees the task even
> get acknowledged by the remote system. It just stays at:
>
> This action will be executed after 1/27/15 10:28:00 AM CST
> This action's status is: Queued.
> This action has not yet been picked up.
>
> Of course, brand new VMs I set up to try to replicate this issue work fine.
>
> Paulo: That's an idea, but not necessarily a *good* one... :)
>
> On Tue, Jan 27, 2015 at 10:29 AM, Paulo Silva <paulojjs at gmail.com> wrote:
>
>> I've found an ugly fix for this, all I had to do was to use rhnreg_ks to
>> re-register the system and now events on the new  profile are being picked
>> up by rhn_check.
>>
>> 2015-01-27 16:07 GMT+00:00 Paulo Silva <paulojjs at gmail.com>:
>>
>>> Hi Glen,
>>>
>>> Can't try that since I'm using rhnsd and not osad.
>>>
>>> Thanks
>>>
>>> 2015-01-27 15:28 GMT+00:00 Glen Collins <glenc2004 at comcast.net>:
>>>
>>>> Hi Paulo,
>>>>
>>>>    Why don't you try this on the client....
>>>>
>>>> service osad stop
>>>> rm -rf /etc/sysconfig/rhn/osad-auth.conf
>>>> service osad start
>>>> (Wait for 2 minutes) and run (On the client)
>>>> /usr/sbin/rhn_check
>>>>
>>>> I've had the same issues and just restarting OSAD didn't do it for me.
>>>> But when I removed the auth file, that seemed to fix things. Not elegant in
>>>> any sort of means but it seems to work. Have not tried it long term since
>>>> I'm putting this "fix" into place right now. I guess this is what happens
>>>> when you cobble a product together and "make it work". I have found jabber
>>>> to be a PITA!
>>>>
>>>> Anyways, hope this helps!
>>>>
>>>> Glen Collins
>>>>
>>>> ------------------------------
>>>> It's not permissions:
>>>>
>>>> [root at zbx-prx-elk-04 ~]# rhn-actions-control --report
>>>> deploy is enabled
>>>> diff is enabled
>>>> upload is enabled
>>>> mtime_upload is enabled
>>>> run is enabled
>>>>
>>>> It's not SSL either, we have our systems configured to speak in the
>>>> clear while troubleshooting this issue. Flipping over to HTTPS doesn't
>>>> change anything.
>>>>
>>>> This particular server has three tasks pending now: two config file
>>>> pushes and one remote command (just to run "date").  Here's the output of
>>>> rhn_check -vvvv:
>>>>
>>>> [root at zbx-prx-elk-04 ~]# rhn_check -vvvv
>>>> D: opening  db environment /var/lib/rpm cdb:mpool:joinenv
>>>> D: opening  db index       /var/lib/rpm/Packages rdonly mode=0x0
>>>> D: locked   db index       /var/lib/rpm/Packages
>>>> D: loading keyring from pubkeys in /var/lib/rpm/pubkeys/*.key
>>>> D: couldn't find any keys in /var/lib/rpm/pubkeys/*.key
>>>> D: loading keyring from rpmdb
>>>> D: opening  db index       /var/lib/rpm/Name rdonly mode=0x0
>>>> D: added key gpg-pubkey-b6f8c6bf-48853124 to keyring
>>>> D: added key gpg-pubkey-8db2536f-4ade0b9d to keyring
>>>> D: added key gpg-pubkey-66fd4949-4803fe57 to keyring
>>>> D: added key gpg-pubkey-c105b9de-4e0fd3a3 to keyring
>>>> D: added key gpg-pubkey-217521f6-45e8a532 to keyring
>>>> D: added key gpg-pubkey-0608b895-4bd22942 to keyring
>>>> D: added key gpg-pubkey-863a853d-4f55f54d to keyring
>>>> D: added key gpg-pubkey-79ea5ed4-508d25a6 to keyring
>>>> D: Using legacy gpg-pubkey(s) from rpmdb
>>>> D: opening  db index       /var/lib/rpm/Providename rdonly mode=0x0
>>>> D: do_call packages.checkNeedUpdate('rhnsd=1',){}
>>>> D: opening  db environment /var/lib/rpm cdb:mpool:joinenv
>>>> D: opening  db index       /var/lib/rpm/Packages rdonly mode=0x0
>>>> D: loading keyring from pubkeys in /var/lib/rpm/pubkeys/*.key
>>>> D: couldn't find any keys in /var/lib/rpm/pubkeys/*.key
>>>> D: loading keyring from rpmdb
>>>> D: opening  db index       /var/lib/rpm/Name rdonly mode=0x0
>>>> D: added key gpg-pubkey-b6f8c6bf-48853124 to keyring
>>>> D: added key gpg-pubkey-8db2536f-4ade0b9d to keyring
>>>> D: added key gpg-pubkey-66fd4949-4803fe57 to keyring
>>>> D: added key gpg-pubkey-c105b9de-4e0fd3a3 to keyring
>>>> D: added key gpg-pubkey-217521f6-45e8a532 to keyring
>>>> D: added key gpg-pubkey-0608b895-4bd22942 to keyring
>>>> D: added key gpg-pubkey-863a853d-4f55f54d to keyring
>>>> D: added key gpg-pubkey-79ea5ed4-508d25a6 to keyring
>>>> D: Using legacy gpg-pubkey(s) from rpmdb
>>>> D: opening  db index       /var/lib/rpm/Providename rdonly mode=0x0
>>>> D: closed   db index       /var/lib/rpm/Providename
>>>> D: closed   db index       /var/lib/rpm/Name
>>>> D: closed   db index       /var/lib/rpm/Packages
>>>> D: closed   db environment /var/lib/rpm
>>>> Loaded plugins: fastestmirror, rhnplugin
>>>> Config time: 0.040
>>>> D: login(forceUpdate=False) invoked
>>>> D: readCachedLogin invoked
>>>> D: Checking pickled loginInfo, currentTime=1422370609.99,
>>>> createTime=1422370587.99, expire-offset=3600.0
>>>> D: readCachedLogin(): using pickled loginInfo set to expire at
>>>> 1422374187.99
>>>> D: rpcServer: Calling XMLRPC up2date.listChannels
>>>> Setting up Package Sacks
>>>> Loading mirror speeds from cached hostfile
>>>> pkgsack time: 0.071
>>>> rpmdb time: 0.000
>>>> Loading mirror speeds from cached hostfile
>>>> repo time: 0.000
>>>> D: local action status: (0, 'rpm database not modified since last
>>>> update (or package list recently updated)', {})
>>>> D: rpcServer: Calling XMLRPC registration.welcome_message
>>>> D: closed   db index       /var/lib/rpm/Providename
>>>> D: closed   db index       /var/lib/rpm/Name
>>>> D: closed   db index       /var/lib/rpm/Packages
>>>> D: closed   db environment /var/lib/rpm
>>>>
>>>> Paulo: To answer your question about the API, spacecmd calls it
>>>> extensively; and you can look up pending tasks by ID.
>>>>
>>>> spacecmd {SSM:0}> schedule_details 578593
>>>> ID:        578593
>>>> Action:    Remote Command on zbx-prx-elk-04.company.com.
>>>> User:      admincswingler
>>>> Date:      20150126T16:36:00
>>>>
>>>> Completed:   0
>>>> Failed:      0
>>>> Pending:     1
>>>>
>>>> Pending Systems
>>>> ---------------
>>>> zbx-prx-elk-04.company.com
>>>> spacecmd {SSM:0}> schedule_details 578588
>>>> ID:        578588
>>>> Action:    Deploy config files to system
>>>> User:      admincswingler
>>>> Date:      20150126T16:22:00
>>>>
>>>> Completed:   0
>>>> Failed:      0
>>>> Pending:     1
>>>>
>>>> Pending Systems
>>>> ---------------
>>>> zbx-prx-elk-04.company.com
>>>> spacecmd {SSM:0}> schedule_details 578586
>>>> ID:        578586
>>>> Action:    Deploy config files to system
>>>> User:      admincswingler
>>>> Date:      20150126T16:18:00
>>>>
>>>> Completed:   0
>>>> Failed:      0
>>>> Pending:     1
>>>>
>>>> Pending Systems
>>>> ---------------
>>>> zbx-prx-elk-04.company.com
>>>> spacecmd {SSM:0}>
>>>>
>>>> Still scratching my head on this one, and getting rather frustrated :)
>>>>
>>>> On Tue, Jan 27, 2015 at 7:02 AM, Paulo Silva <paulojjs at gmail.com>
>>>> wrote:
>>>>
>>>>> I'm checking pending events using the web interface (URL
>>>>> /rhn/systems/details/history/Pending.do?sid=1000010296) and it shows 2
>>>>> pending events: and Hardware Refresh and an Errata Update.
>>>>>
>>>>> The sid seams to be correct with the server I'm running my commands:
>>>>>
>>>>> # grep 1000010296 /etc/sysconfig/rhn/systemid
>>>>> <value><string>ID-1000010296</string></value>
>>>>>
>>>>> Is there a way to check for pending events using the API?
>>>>>
>>>>> 2015-01-27 12:45 GMT+00:00 Alexander Innes <senni at necurity.co.uk>:
>>>>>
>>>>>> Just to make sure im not thinking of the wrong feture are you trying
>>>>>> to use to remote commands?
>>>>>> Systems -> System -> somethin -> remote command (or a different one
>>>>>> so i can test in our environment :) )
>>>>>>
>>>>>> from that output its not seeing anything as you said :o
>>>>>>
>>>>>> On 27 January 2015 at 12:16, Paulo Silva <paulojjs at gmail.com> wrote:
>>>>>>
>>>>>>> I'm having the same issue, running rhn_check does this:
>>>>>>>
>>>>>>> # rhn_check -vvv
>>>>>>> D: opening  db environment /var/lib/rpm cdb:mpool:joinenv
>>>>>>> D: opening  db index       /var/lib/rpm/Packages rdonly mode=0x0
>>>>>>> D: locked   db index       /var/lib/rpm/Packages
>>>>>>> D: loading keyring from pubkeys in /var/lib/rpm/pubkeys/*.key
>>>>>>> D: couldn't find any keys in /var/lib/rpm/pubkeys/*.key
>>>>>>> D: loading keyring from rpmdb
>>>>>>> D: opening  db index       /var/lib/rpm/Name rdonly mode=0x0
>>>>>>> D: added key gpg-pubkey-c105b9de-4e0fd3a3 to keyring
>>>>>>> D: added key gpg-pubkey-b5c61460-41667588 to keyring
>>>>>>> D: added key gpg-pubkey-6b8d79e6-3f49313d to keyring
>>>>>>> D: added key gpg-pubkey-0608b895-4bd22942 to keyring
>>>>>>> D: added key gpg-pubkey-863a853d-4f55f54d to keyring
>>>>>>> D: Using legacy gpg-pubkey(s) from rpmdb
>>>>>>> D: opening  db index       /var/lib/rpm/Providename rdonly mode=0x0
>>>>>>> D: do_call packages.checkNeedUpdate('rhnsd=1',){}
>>>>>>> D: opening  db environment /var/lib/rpm cdb:mpool:joinenv
>>>>>>> D: opening  db index       /var/lib/rpm/Packages rdonly mode=0x0
>>>>>>> D: loading keyring from pubkeys in /var/lib/rpm/pubkeys/*.key
>>>>>>> D: couldn't find any keys in /var/lib/rpm/pubkeys/*.key
>>>>>>> D: loading keyring from rpmdb
>>>>>>> D: opening  db index       /var/lib/rpm/Name rdonly mode=0x0
>>>>>>> D: added key gpg-pubkey-c105b9de-4e0fd3a3 to keyring
>>>>>>> D: added key gpg-pubkey-b5c61460-41667588 to keyring
>>>>>>> D: added key gpg-pubkey-6b8d79e6-3f49313d to keyring
>>>>>>> D: added key gpg-pubkey-0608b895-4bd22942 to keyring
>>>>>>> D: added key gpg-pubkey-863a853d-4f55f54d to keyring
>>>>>>> D: Using legacy gpg-pubkey(s) from rpmdb
>>>>>>> D: opening  db index       /var/lib/rpm/Providename rdonly mode=0x0
>>>>>>> D: closed   db index       /var/lib/rpm/Providename
>>>>>>> D: closed   db index       /var/lib/rpm/Name
>>>>>>> D: closed   db index       /var/lib/rpm/Packages
>>>>>>> D: closed   db environment /var/lib/rpm
>>>>>>> Loaded plugins: etckeeper, fastestmirror, rhnplugin
>>>>>>> Config time: 0.073
>>>>>>> D: login(forceUpdate=False) invoked
>>>>>>> D: readCachedLogin invoked
>>>>>>> D: Checking pickled loginInfo, currentTime=1422360920.07,
>>>>>>> createTime=1422360336.74, expire-offset=3600.0
>>>>>>> D: readCachedLogin(): using pickled loginInfo set to expire at
>>>>>>> 1422363936.74
>>>>>>> D: rpcServer: Calling XMLRPC up2date.listChannels
>>>>>>> This system is receiving updates from RHN Classic or Red Hat
>>>>>>> Satellite.
>>>>>>> rpmdb time: 0.000
>>>>>>> Loading mirror speeds from cached hostfile
>>>>>>> repo time: 0.002
>>>>>>> Setting up Package Sacks
>>>>>>> pkgsack time: 0.253
>>>>>>> D: local action status: (0, 'rpm database not modified since last
>>>>>>> update (or package list recently updated)', {})
>>>>>>> D: rpcServer: Calling XMLRPC registration.welcome_message
>>>>>>> D: closed   db index       /var/lib/rpm/Providename
>>>>>>> D: closed   db index       /var/lib/rpm/Name
>>>>>>> D: closed   db index       /var/lib/rpm/Packages
>>>>>>> D: closed   db environment /var/lib/rpm
>>>>>>>
>>>>>>>
>>>>>>> 2015-01-27 10:53 GMT+00:00 Alexander Innes <senni at necurity.co.uk>:
>>>>>>>
>>>>>>>> Schedual the action then on the server do rhn-chck -vvv. It will
>>>>>>>> tell you what the client's doing, my gut feeling is either SSL errors OR
>>>>>>>> permsions (rhn-actions-control is it?)
>>>>>>>>
>>>>>>>> On 26 January 2015 at 23:16, Chris Swingler <
>>>>>>>> chris at chrisswingler.com> wrote:
>>>>>>>>
>>>>>>>>> Hey Spacewalk-list!
>>>>>>>>>
>>>>>>>>> I have quite a few systems (I haven't dug into seeing if there is
>>>>>>>>> any consistency in CentOS versions or the RHN tools) that just plain do not
>>>>>>>>> seem to get tasks back from the Spacewalk server.  We're running Spacewalk
>>>>>>>>> 2.2 on CentOS release 6.5 (Final).
>>>>>>>>>
>>>>>>>>> I can create a simple task in Spacewalk, like just print the
>>>>>>>>> output of "date", and an rhn_check never seems to execute it.  It shows up
>>>>>>>>> under Events > Pending, the SystemIDs match, but it never moves.
>>>>>>>>>
>>>>>>>>> Running a packet capture to watch the transaction, it looks like
>>>>>>>>> Spacewalk itself isn't even sending the task to the remote system.
>>>>>>>>> Spacewalk happily replies with an HTTP 200 and an empty payload.
>>>>>>>>>
>>>>>>>>> HTTP/1.1 200 OK
>>>>>>>>> Date: Mon, 26 Jan 2015 22:47:55 GMT
>>>>>>>>> Server: Apache
>>>>>>>>> Content-Length: 126
>>>>>>>>> X-RHN-Server-Capability: registration.finish_message(1)=1
>>>>>>>>> X-RHN-Server-Capability: registration.remaining_subscriptions(1)=1
>>>>>>>>> X-RHN-Server-Capability: abrt(1)=1
>>>>>>>>> X-RHN-Server-Capability: registration.update_contact_info(1)=1
>>>>>>>>> X-RHN-Server-Capability: staging_content(1)=1
>>>>>>>>> X-RHN-Server-Capability: applet.has_base_channel(1)=1
>>>>>>>>> X-RHN-Server-Capability: registration.smbios(1)=1
>>>>>>>>> X-RHN-Server-Capability: registration.extended_update_support(1)=1
>>>>>>>>> X-RHN-Server-Capability: rhncfg.filetype.directory(1)=1
>>>>>>>>> X-RHN-Server-Capability: registration.update_systemid(1)=1
>>>>>>>>> X-RHN-Server-Capability: registration.register_osad(1)=1
>>>>>>>>> X-RHN-Server-Capability: registration.delta_packages(1)=1
>>>>>>>>> X-RHN-Server-Capability: cpu_sockets(1)=1
>>>>>>>>> X-RHN-Server-Capability: ipv6(1)=1
>>>>>>>>> X-RHN-Server-Capability: rhncfg.content.base64_decode(1)=1
>>>>>>>>> X-RHN-Server-Capability: xmlrpc.packages.extended_profile(1-2)=1
>>>>>>>>> X-RHN-Server-Capability: xmlrpc.errata.patch_names(1)=1
>>>>>>>>> X-RHN-Server-Capability: xmlrpc.packages.checksums(1)=1
>>>>>>>>> X-RHN-Server-Capability: xmlrpc.login.extra_data(1)=1
>>>>>>>>> X-RHN-Proxy-Version: 0
>>>>>>>>> X-Transport-Info: Extended Capabilities Transport (C) Red Hat, Inc (version 2.5.72-1.el6)
>>>>>>>>> X-RHN-Client-Version: 1
>>>>>>>>> Connection: close
>>>>>>>>> Content-Type: text/xml
>>>>>>>>>
>>>>>>>>> <?xml version='1.0'?>
>>>>>>>>> <methodResponse>
>>>>>>>>> <params>
>>>>>>>>> <param>
>>>>>>>>> <value><string></string></value>
>>>>>>>>> </param>
>>>>>>>>> </params>
>>>>>>>>> </methodResponse>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Full transaction at
>>>>>>>>>
>>>>>>>>> https://gist.github.com/cswingler/f718abcb9ce290adec29
>>>>>>>>>
>>>>>>>>> rhn_server_xmlrpc.log simply shows:
>>>>>>>>>
>>>>>>>>> 2015/01/26 16:52:07 -05:00 30997 172.29.7.14:
>>>>>>>>> xmlrpc/queue.get(1000015056, 2, 'checkins enabled')
>>>>>>>>> 2015/01/26 16:52:07 -05:00 30998 172.29.7.14:
>>>>>>>>> xmlrpc/up2date.listChannels(1000015056,)
>>>>>>>>> 2015/01/26 16:52:08 -05:00 30995 172.29.7.14:
>>>>>>>>> xmlrpc/registration.welcome_message('lang: None',)
>>>>>>>>>
>>>>>>>>> Any basic things I should be checking? I've tried updating the
>>>>>>>>> spacewalk/rhn tools on the agent, restarting Spacewalk itself, and nothing
>>>>>>>>> seems to make a change. This issue, as far as I can recall, did not seem to
>>>>>>>>> appear until after we upgraded to 2.2.
>>>>>>>>>
>>>>>>>>> Some systems seem just fine, and I can't seem to find a consistent
>>>>>>>>> reason why some succeed and others fail.
>>>>>>>>>
>>>>>>>>> Thanks!
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Spacewalk-list mailing list
>>>>>>>>> Spacewalk-list at redhat.com
>>>>>>>>> https://www.redhat.com/mailman/listinfo/spacewalk-list
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Spacewalk-list mailing list
>>>>>>>> Spacewalk-list at redhat.com
>>>>>>>> https://www.redhat.com/mailman/listinfo/spacewalk-list
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Paulo Silva <paulojjs at gmail.com>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Spacewalk-list mailing list
>>>>>>> Spacewalk-list at redhat.com
>>>>>>> https://www.redhat.com/mailman/listinfo/spacewalk-list
>>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Spacewalk-list mailing list
>>>>>> Spacewalk-list at redhat.com
>>>>>> https://www.redhat.com/mailman/listinfo/spacewalk-list
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Paulo Silva <paulojjs at gmail.com>
>>>>>
>>>>> _______________________________________________
>>>>> Spacewalk-list mailing list
>>>>> Spacewalk-list at redhat.com
>>>>> https://www.redhat.com/mailman/listinfo/spacewalk-list
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Spacewalk-list mailing list
>>>> Spacewalk-list at redhat.com
>>>> https://www.redhat.com/mailman/listinfo/spacewalk-list
>>>>
>>>>
>>>> _______________________________________________
>>>> Spacewalk-list mailing list
>>>> Spacewalk-list at redhat.com
>>>> https://www.redhat.com/mailman/listinfo/spacewalk-list
>>>>
>>>
>>>
>>>
>>> --
>>> Paulo Silva <paulojjs at gmail.com>
>>>
>>
>>
>>
>> --
>> Paulo Silva <paulojjs at gmail.com>
>>
>> _______________________________________________
>> Spacewalk-list mailing list
>> Spacewalk-list at redhat.com
>> https://www.redhat.com/mailman/listinfo/spacewalk-list
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/spacewalk-list/attachments/20150127/2af3107e/attachment.htm>

More information about the Spacewalk-list mailing list