[Spacewalk-list] Antwort: Re: Errata Cache bunch running forever on Spacewalk 2.4

Tue Jan 12 08:08:47 UTC 2016

Hello again,

still having problems,

I noticed that the errata-cache gets triggerd when a client refreshes its 
repos or searches for updates. So I manually fired a "zypper ref -s" or a 
"apt-get update" on every client connected to this spacewalk server. This 
seems to have fixed "something". Suddenly, the "errata-cache" task has 
finished and I got some backtraces from spacewalk.

But I still have trouble. The UI does not seem to get updated correctly 
(on the "system" page, nearly all systems look "up-to-date". But I know, 
it's not!). But the updates are available to the clients (searching on the 
client).

When I look into the last errata-cache schedules (in Spacewalk "Admin" -> 
"Task schedules" -> "errata-cache-bunch") I can see, that the 
"errata-cache" tasks is often "SKIPPED"

errata-cache 
2016-01-12 08:50:00 MEZ 
2016-01-12 08:50:00 MEZ 
SKIPPED 
errata-cache 
2016-01-12 08:49:00 MEZ 
2016-01-12 08:49:00 MEZ 
SKIPPED 
errata-cache 
2016-01-12 08:48:00 MEZ 
2016-01-12 08:48:00 MEZ 
SKIPPED 
errata-cache 
2016-01-12 08:47:00 MEZ 
2016-01-12 08:50:00 MEZ 
FINISHED 
errata-cache 
2016-01-12 08:46:00 MEZ 
2016-01-12 08:46:00 MEZ 
SKIPPED 
errata-cache 
2016-01-12 08:45:00 MEZ 
2016-01-12 08:45:00 MEZ 
SKIPPED 
errata-cache 
2016-01-12 08:44:00 MEZ 
2016-01-12 08:46:00 MEZ 
FINISHED 
errata-mailer 
2016-01-12 08:43:00 MEZ 
2016-01-12 08:43:00 MEZ 
FINISHED 
errata-cache 
2016-01-12 08:43:00 MEZ 
2016-01-12 08:43:00 MEZ 
FINISHED 
errata-cache 
2016-01-12 08:42:00 MEZ 
2016-01-12 08:42:00 MEZ 
SKIPPED 
errata-cache 
2016-01-12 08:41:00 MEZ 
2016-01-12 08:41:00 MEZ 
SKIPPED 
errata-cache 
2016-01-12 08:40:00 MEZ 
2016-01-12 08:40:00 MEZ 
SKIPPED 
errata-cache 
2016-01-12 08:39:00 MEZ 
2016-01-12 08:42:00 MEZ 
FINISHED 
errata-mailer 
2016-01-12 08:38:00 MEZ 
2016-01-12 08:38:00 MEZ 
FINISHED 
errata-cache 
2016-01-12 08:38:00 MEZ 
2016-01-12 08:38:00 MEZ 
FINISHED 
errata-mailer 
2016-01-12 08:37:00 MEZ 
2016-01-12 08:37:00 MEZ 
FINISHED 
errata-cache 
2016-01-12 08:37:00 MEZ 
2016-01-12 08:37:00 MEZ 
FINISHED 
errata-cache 
2016-01-12 08:36:00 MEZ 
2016-01-12 08:36:00 MEZ 
SKIPPED 
errata-cache 
2016-01-12 08:35:00 MEZ 
2016-01-12 08:35:00 MEZ 
SKIPPED 
errata-cache 
2016-01-12 08:34:00 MEZ 
2016-01-12 08:36:00 MEZ 
FINISHED 
errata-mailer 
2016-01-12 08:33:00 MEZ 
2016-01-12 08:33:00 MEZ 
FINISHED 
errata-cache 
2016-01-12 08:33:00 MEZ 
2016-01-12 08:33:00 MEZ 
FINISHED 
errata-mailer 
2016-01-12 08:32:00 MEZ 
2016-01-12 08:32:00 MEZ 
FINISHED 
errata-cache 
2016-01-12 08:32:00 MEZ 
2016-01-12 08:32:00 MEZ 
FINISHED 
errata-mailer 
2016-01-12 08:31:00 MEZ 
2016-01-12 08:31:00 MEZ 
FINISHED 
errata-cache 
2016-01-12 08:31:00 MEZ 
2016-01-12 08:31:00 MEZ 
FINISHED 
errata-mailer 
2016-01-12 08:30:00 MEZ 
2016-01-12 08:30:00 MEZ 
FINISHED 
errata-cache 
2016-01-12 08:30:00 MEZ 
2016-01-12 08:30:00 MEZ 
FINISHED 
errata-cache 
2016-01-12 08:29:00 MEZ 
2016-01-12 08:29:00 MEZ 
SKIPPED 

And also I noticed that the "errata-mailer" tasks seems to get only 
started, when the "errata-cache" tasks finished within one minute. (Just 
look above). As soon as the errata-cache tasks runs longer than one 
minute, the "errata-mailer" is not started. I'm also not sure, if 
something went wrong, while I was shutting down the spacewalk server for 
the update.

This is what it looked before (on spacewalk 2.2). I started the update on 
2016-01-04 abount 08:30.

errata-cache 
2016-01-04 10:10:00 MEZ 
2016-01-04 10:10:00 MEZ 
SKIPPED 
errata-cache 
2016-01-04 10:00:00 MEZ 
2016-01-04 10:00:00 MEZ 
SKIPPED 
errata-cache 
2016-01-04 09:50:00 MEZ 
2016-01-04 09:50:00 MEZ 
SKIPPED 
errata-cache 
2016-01-04 09:40:00 MEZ 
2016-01-04 09:40:00 MEZ 
SKIPPED 
errata-cache 
2016-01-04 09:30:00 MEZ 
2016-01-04 09:30:00 MEZ 
SKIPPED 
errata-cache 
2016-01-04 09:20:00 MEZ 
2016-01-04 09:20:00 MEZ 
SKIPPED 
errata-cache 
2016-01-04 09:10:00 MEZ 
2016-01-04 09:10:00 MEZ 
SKIPPED 
errata-cache 
2016-01-04 09:00:00 MEZ 
2016-01-04 09:00:00 MEZ 
SKIPPED 
errata-cache 
2016-01-04 08:50:00 MEZ 
2016-01-04 08:50:00 MEZ 
SKIPPED 
errata-cache 
2016-01-04 08:40:00 MEZ 
2016-01-04 08:40:00 MEZ 
SKIPPED 
errata-cache 
2016-01-04 08:30:00 MEZ 
2016-01-05 08:59:59 MEZ 
INTERRUPTED 
errata-mailer 
2016-01-04 08:30:00 MEZ 
2016-01-04 08:29:57 MEZ 
INTERRUPTED 
errata-mailer 
2016-01-04 08:00:00 MEZ 
2016-01-04 08:00:00 MEZ 
FINISHED 
errata-cache 
2016-01-04 08:00:00 MEZ 
2016-01-04 08:00:00 MEZ 
FINISHED 
errata-mailer 
2016-01-04 07:50:00 MEZ 
2016-01-04 07:50:00 MEZ 
FINISHED 
errata-cache 
2016-01-04 07:50:00 MEZ 
2016-01-04 07:50:00 MEZ 
FINISHED 

I would be glad, if someone could give me further hints, where to look for 
the root cause to this.

Kind regards,
Robert

Mit freundlichen Grüßen

Robert Paschedag
Netlution GmbH
Landteilstr. 33
68163 Mannheim

im Auftrag des
SWR
Südwestrundfunk
Informations- und Kommunikationssysteme
Neckarstraße 230
70190 Stuttgart

Telefon +49 (0)711 /929-12654 oder
Telefon +49 (0)711 /929-13714
paschedag.netlution at swr.de

swr.de

Von:    Robert Paschedag <robert.paschedag at web.de>
An:     Paschedag.Netlution at swr.de
Kopie:  spacewalk-list at redhat.com
Datum:  08.01.2016 21:53
Betreff:        Re: [Spacewalk-list] Errata Cache bunch running forever on 
Spacewalk 2.4

Hi again,

just some further information.

Because I already activated debug logging of taskomatic, searching for
server errata cache tasks within the rhn_taskomatic_daemon logfiles with

grep 'Updating errata cache for server' rhn_taskomatic_daemon.log* | awk
'{print $NF; }' | sort | uniq -c

i get

    14 [1000010274]
     14 [1000010299]
      4 [1000010504]
     14 [1000010512]
     14 [1000010516]
     14 [1000010517]
     14 [1000010518]
     14 [1000010519]
     14 [1000010520]
     14 [1000010521]
     14 [1000010522]
     14 [1000010523]
     14 [1000010524]
     14 [1000010527]
     14 [1000010528]
     14 [1000010529]
     14 [1000010530]
     14 [1000010531]
     14 [1000010532]
     14 [1000010564]
     14 [1000010570]
     15 [1000010573]
     15 [1000010593]
     13 [1000010639]
     14 [1000010640]
     15 [1000010641]
     13 [1000010642]
     14 [1000010647]
     13 [1000010648]
     14 [1000010649]
     15 [1000010650]
     14 [1000010651]
     14 [1000010652]
      6 [1000010672]
     15 [1000010683]
     14 [1000010684]
     14 [1000010685]
     14 [1000010687]
     14 [1000010688]
     14 [1000010689]
     14 [1000010690]
     14 [1000010691]
     14 [1000010692]
     14 [1000010693]
     14 [1000010694]
     15 [1000010695]
     14 [1000010697]
     14 [1000010698]
      2 [1000010702]
     14 [1000010705]
     14 [1000010710]
      2 [1000010715]

On this server, there are more than 400 servers registered, so this
looks like, something is "hanging" on these servers and not going any
further (i think).

But right now, I don't know - exactly - where to look for this "error"
or within which DB table to further track this down.

Regards,
Robert

Am 08.01.2016 um 19:31 schrieb Robert Paschedag:
> Can anybody give a hint, in which db table to search for these 
errata-cache tasks, that are started over and over again? I just don't get 
it myself right now . It also looks like that it is always the same prox. 
50 servers that this occurs.
> 
> Regards
> Robert
> Am 07.01.2016 10:11 vorm. schrieb Paschedag.Netlution at swr.de:
>>
>> Hi all, 
>>
>> it looks like one (or more?) jobs have been interrupted, while the 
spacewalk service had shut down for upgrade. I upgraded my spacewalk 
server from 2.2 -> 2.3 -> 2.4. 
>>
>> Now it looks like the errata-cache bunch tasks is running forever and 
the errata notification mail is not started anymore. I think it gets 
called from errata-cache, when this has finished. 
>>
>> Errata Cache:
>> 2016-01-07 10:00:00 MEZ
>> RUNNING
>> Errata Notification Mail:
>> 2016-01-04 08:30:00 MEZ
>> INTERRUPTED
>>
>>
>> When I look into the database, I can see several tasks in the 
rhntaskqueue but I'm pretty sure, that these tasks do not get cleared 
correctly, so the task is run for every server over and over again. I'm 
not sure! This is just my thinking. 
>>
>> rhnschema=# select * from rhntaskqueue; 
>>  org_id |         task_name          | task_data  | priority |  
earliest 
>> 
--------+----------------------------+------------+----------+------------------------------- 

>>       1 | update_server_errata_cache | 1000010701 |        0 | 
2016-01-07 10:00:45.963006+01 
>>       1 | update_server_errata_cache | 1000010274 |        0 | 
2016-01-07 10:00:53.198001+01 
>>       1 | update_server_errata_cache | 1000010527 |        0 | 
2016-01-07 10:01:03.768592+01 
>>       1 | update_server_errata_cache | 1000010689 |        0 | 
2016-01-07 10:01:56.915296+01 
>>       1 | update_server_errata_cache | 1000010641 |        0 | 
2016-01-07 10:02:41.775593+01 
>>       1 | update_server_errata_cache | 1000010687 |        0 | 
2016-01-07 10:05:01.471365+01 
>>       1 | update_server_errata_cache | 1000010650 |        0 | 
2016-01-07 10:05:26.492405+01 
>>       1 | update_server_errata_cache | 1000010520 |        0 | 
2016-01-07 10:06:38.052352+01 
>> (8 Zeilen) 
>>
>> rhnschema=# 
>>
>> Any help will be appreciated to debug this further. 
>>
>> Regards, 
>> Robert 
>>
>>
>> Mit freundlichen Grüßen 
>>
>> Robert Paschedag 
>> Netlution GmbH 
>> Landteilstr. 33 
>> 68163 Mannheim 
>>
>> im Auftrag des 
>> SWR 
>> Südwestrundfunk 
>> Informations- und Kommunikationssysteme 
>> Neckarstraße 230 
>> 70190 Stuttgart 
>>
>> Telefon +49 (0)711 /929-12654 oder 
>> Telefon +49 (0)711 /929-13714 
>> paschedag.netlution at swr.de 
>>
>> swr.de 
>>
> 
> _______________________________________________
> Spacewalk-list mailing list
> Spacewalk-list at redhat.com
> https://www.redhat.com/mailman/listinfo/spacewalk-list
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/spacewalk-list/attachments/20160112/a059b356/attachment.htm>