[Spacewalk-list] Taskomatic runs indefinitely without ever generating repodata
Gerald Vogt
vogt at spamcop.net
Wed Jul 18 13:19:08 UTC 2018
On 14.07.18 16:07, Paul Dias - BCX wrote:
> Hi,
>
>
> you can put tomcat in debug mode and run tomcat if I remember without
> calling the service which forks it into the background, but actually
> display it in the console, that way you can see what is happening when
I don't think taskomatic is a tomcat. I can run taskomatic on the
console. But that only prints all the log lines on stdout and nothing
further.
> you run a job, also in /usr/share/rhn/config-defaults/rhn_taskomatic,
> there are options there that you can increase the level of logging from
> what I can see. I cant remember clearly but there was a post that I was
> going through a couple of weeks ago about troubleshooting I can't
> remember the address tbh!
Even with DEBUG log level it doesn't show anything beyond that it starts
the channel-repodata task but never finishes... I have no idea what it
is actually doing there...
I am stumped. Currently, we are unable to update our spacewalk server...
Thanks,
Gerald
>
>
> Regards,
> *Paul Dias*
> 6^th Floor, 8 Boundary Road
> Newlands
> Cape Town
> 7700
> T: +27 (0) 21 681 3149
>
> *Meet your future today.*
> **
> BCX
>
>
> ------------------------------------------------------------------------
> *From:* Gerald Vogt <vogt at spamcop.net>
> *Sent:* Friday, 13 July 2018 8:56 AM
> *To:* spacewalk-list at redhat.com
> *Subject:* Re: [Spacewalk-list] Taskomatic runs indefinitely without
> ever generating repodata
> Anyone any idea how to troubleshoot this? Any debug logging we could
> enable to find out what's really going on and where it's hanging?
>
> Thanks,
>
> Gerald
>
> On 06.07.18 09:20, Gerald Vogt wrote:
>> On 05.07.18 21:30, Matt Moldvan wrote:
>>> Is there anything interesting
>>> in /var/log/rhn/tasko/sat/channel-repodata-bunch? Do you have any hung
>>
>> There is currently only a single file with this content:
>>
>> spacewalk:channel-repodata-bunch(996)# ls -l
>> total 4
>> -rw-r--r--. 1 root root 130 Jul 2 08:13 channel-repodata_15408487_out
>> spacewalk:channel-repodata-bunch(997)# cat channel-repodata_15408487_out
>> 2018-07-02 08:13:10,793 [DefaultQuartzScheduler_Worker-8] INFO
>> com.redhat.rhn.taskomatic.task.ChannelRepodata - In the queue: 4
>>
>>> reposync processes? Any lingering Postgres locks that might be an issue?
>>
>> No reposync processes. All progres processes say "idle", so I guess
>> there are no locks. Or how do I check for lingering locks?
>>
>>> It's odd that the run would only take 1 second, unless something is
>>> wrong with the database or it's data...
>>>
>>> What do you see from a spacewalk-sql command like below?
>>
>> I see all the channels:
>>
>> label |
>> name | modified |
>> last_synced
>> -----------------------------------+------------------------------------------------+-------------------------------+----------------------------
>>
>> icinga-epel7-x86_64 | ICINGA stable release for epel-7
>> (x86_64) | 2016-02-15 10:07:59.822942+01 | 2018-07-06 02:30:55.482+02
>> epel7-centos7-x86_64 | EPEL 7 for CentOS 7
>> (x86_64) | 2014-07-21 08:16:26.367135+02 | 2018-07-06
>> 04:01:52.148+02
>> centos6-x86_64-extras | CentOS 6 Extras
>> (x86_64) | 2012-08-23 06:46:05.145629+02 |
>> 2018-06-21 10:25:26.104+02
>> grafana-epe7-x86_64 | Grafana stable release for epel-7
>> (x86_64) | 2016-05-06 08:29:49.308149+02 | 2018-06-21 04:58:15.022+02
>> spacewalk26-client-centos6-x86_64 | Spacewalk Client 2.6 for CentOS 6
>> (x86_64) | 2017-04-25 13:44:49.266738+02 | 2018-06-21 10:41:07.369+02
>> globus-el6-x86_64 | Globus Toolkit 6
>> (el6) | 2016-05-13 15:23:31.807011+02 |
>> 2018-07-06 03:34:49.95+02
>> internet2 | perfSONAR RPM
>> Repository | 2017-06-27 06:56:33.675378+02 |
>> 2018-06-22 10:24:41.702+02
>> postgresql94-centos6-x86_64 | PostgreSQL 9.4 for CentOS 6
>> (x86_64) | 2015-01-28 14:09:41.856451+01 | 2018-06-21
>> 10:42:01.413+02
>> spacewalk26-server-centos6-x86_64 | Spacewalk Server 2.6 for CentOS 6
>> (x86_64) | 2017-04-25 13:39:38.250769+02 | 2018-06-21 10:36:17.46+02
>> centos7-x86_64-fasttrack | CentOS 7 FastTrack
>> (x86_64) | 2014-07-21 08:16:26.017642+02 | 2018-06-21
>> 10:26:29.571+02
>> spacewalk26-client-centos7-x86_64 | Spacewalk Client 2.6 for CentOS 7
>> (x86_64) | 2017-04-25 13:46:00.107344+02 | 2018-06-22 10:22:28.484+02
>> centos7-x86_64-centosplus | CentOS 7 Plus
>> (x86_64) | 2014-07-21 08:16:25.467309+02 |
>> 2018-06-21 10:25:19.884+02
>> centos6-x86_64-centosplus | CentOS 6 Plus
>> (x86_64) | 2012-08-23 07:18:00.349338+02 |
>> 2018-06-21 10:36:04.08+02
>> docker-ce-centos7-x86_64 | Docker CE Stable for CentOS 7
>> (x86_64) | 2017-09-28 12:52:45.858354+02 | 2018-07-06
>> 04:30:05.442+02
>> postgresql10-centos7-x86_64 | PostgreSQL 10 for CentOS 7
>> (x86_64) | 2018-02-12 14:48:14.617235+01 | 2018-02-12
>> 15:06:16.464+01
>> bareos162-centos7-x86_64 | Bareos 16.2 for CentOS 7
>> (x86_64) | 2017-09-26 14:37:16.533773+02 | 2018-06-21
>> 04:59:21.954+02
>> docker-ce-edge-centos7-x86_64 | Docker CE Edge for CentOS 7
>> (x86_64) | 2017-12-29 09:58:14.581069+01 | 2018-06-21
>> 04:59:39.796+02
>> beegfs6-centos7-x86_64 | BeeGFS 6 for CentOS 7
>> (x86_64) | 2018-03-19 14:08:08.389588+01 | 2018-06-21
>> 04:59:43.132+02
>> icinga-epel6-x86_64 | ICINGA stable release for epel-6
>> (x86_64) | 2018-01-15 15:41:31.138875+01 | 2018-07-06 02:30:28.142+02
>> openstack-pike-centos7 | OpenStack Pike for CentOS
>> 7 | 2017-10-05 09:10:22.575224+02 | 2018-06-21
>> 05:36:35.43+02
>> globus-el7-x86_64 | Globus Toolkit 6
>> (el7) | 2017-09-28 13:00:07.32028+02 |
>> 2018-07-06 03:31:22.806+02
>> postgresql10-centos6-x86_64 | PostgreSQL 10 for CentOS 6
>> (x86_64) | 2018-02-12 14:48:55.970013+01 | 2018-07-06
>> 04:02:04.03+02
>> ceph-jewel-centos7 | CentOS 7 Ceph Jewel
>> (x86_64) | 2018-02-12 12:15:28.8976+01 | 2018-07-06
>> 05:30:07.085+02
>> spacewalk28-server-centos6-x86_64 | Spacewalk Server 2.8 for CentOS 6
>> (x86_64) | 2018-06-22 18:05:55.190988+02 | 2018-07-06 06:18:15.016+02
>> spacewalk28-client-centos7-x86_64 | Spacewalk Client 2.8 for CentOS 7
>> (x86_64) | 2018-06-22 18:05:55.575963+02 | 2018-07-06 06:18:22.41+02
>> puppet5-el7-x86_64 | Puppet 5 for EL 7
>> (x86_64) | 2018-03-28 14:20:52.254978+02 |
>> 2018-06-21 06:01:31.357+02
>> centos7-qemu-ev | CentOS 7 QEMU EV
>> (x86_64) | 2018-02-12 12:15:06.116673+01 |
>> 2018-07-06 05:30:12.078+02
>> bareos172-centos7-x86_64 | Bareos 17.2 for CentOS 7
>> (x86_64) | 2018-05-08 14:18:56.708206+02 | 2018-06-22
>> 10:24:48.431+02
>> openstack-queens-centos7 | OpenStack Queens for CentOS
>> 7 | 2018-03-28 13:08:27.607498+02 | 2018-07-06
>> 05:30:44.123+02
>> elrepo-centos7 | ELRepo for CentOS
>> 7 | 2017-09-18 12:03:42.302442+02 |
>> 2018-06-21 05:01:30.303+02
>> spacewalk28-client-centos6-x86_64 | Spacewalk Client 2.8 for CentOS 6
>> (x86_64) | 2018-06-22 18:05:53.475193+02 | 2018-07-06 06:18:07.158+02
>> centos7-x86_64-extras | CentOS 7 Extras
>> (x86_64) | 2014-07-21 08:16:25.841121+02 |
>> 2018-06-21 10:26:27.879+02
>> internet2-web100_kernel | perfSONAR Web100 Kernel RPM
>> Repository | 2017-06-27 06:57:03.825602+02 | 2018-06-21
>> 10:24:50.96+02
>> centos6-x86_64-updates | CentOS 6 Updates
>> (x86_64) | 2012-08-23 06:46:05.264195+02 |
>> 2018-06-21 10:34:24.866+02
>> centos7-x86_64-updates | CentOS 7 Updates
>> (x86_64) | 2014-07-21 08:16:26.196397+02 |
>> 2018-07-02 09:48:02.273+02
>> centos6-x86_64-fasttrack | CentOS 6 FastTrack
>> (x86_64) | 2012-08-23 06:46:05.205228+02 | 2018-06-22
>> 10:24:43.51+02
>> postgresql92-centos6-x86_64 | PostgreSQL 9.2 for CentOS 6
>> (x86_64) | 2012-09-12 08:15:27.194188+02 | 2018-07-06
>> 03:47:12.311+02
>> epel6-centos6-x86_64 | EPEL 6 for CentOS 6
>> (x86_64) | 2012-08-23 06:46:30.597753+02 | 2018-07-06
>> 03:55:48.834+02
>> jpackage5.0-generic | JPackage 5.0 for
>> generic | 2014-07-02 10:32:24.985979+02 |
>> 2018-07-06 03:46:46.084+02
>> hp-spp-rhel-7 | HP Software Delivery Repository
>> for SPP RHEL 7 | 2015-04-16 14:18:33.041249+02 | 2018-07-06 05:31:01.633+02
>> owncloud-centos7-noarch | ownCloud for CentOS
>> 7 | 2015-01-28 13:53:41.415573+01 | 2018-06-21
>> 05:36:40.901+02
>> centos7-x86_64-scl | CentOS 7 SCL
>> (x86_64) | 2016-04-15 11:26:29.042925+02 |
>> 2018-06-21 05:14:49.359+02
>> postgresql96-centos6-x86_64 | PostgreSQL 9.6 for CentOS 6
>> (x86_64) | 2017-02-09 15:31:54.632728+01 | 2018-06-21
>> 05:00:39.353+02
>> postgresql96-centos7-x86_64 | PostgreSQL 9.6 for CentOS 7
>> (x86_64) | 2017-02-09 15:35:13.136001+01 | 2018-07-06
>> 04:30:19.645+02
>> centos6-x86_64 | CentOS 6
>> (x86_64) | 2012-08-23 06:46:04.610089+02 |
>> 2018-07-05 22:03:30.089+02
>> centos7-x86_64 | CentOS 7
>> (x86_64) | 2014-07-21 08:16:24.172395+02 |
>> 2018-07-05 22:08:45.242+02
>>
>> -Gerald
>>
>>>
>>> echo 'select label,name,modified,last_synced from rhnchannel' | sudo
>>> spacewalk-sql -i
>>>
>>> label | name | modified|last_synced
>>>
>>> ----------------------------------+----------------------------------+-------------------------------+----------------------------
>>>
>>>
>>> ovirt-x86_64-stable-6-nonprod| ovirt-x86_64-stable-6-nonprod|
>>> 2015-09-14 13:46:44.147134-05 |
>>>
>>> extras7-x86_64-nonprod | extras7-x86_64-nonprod | 2017-11-06
>>> 10:26:30.011283-06 |
>>>
>>> centos7-x86_64-all | centos7-x86_64-all | 2015-11-11
>>> 08:50:58.831234-06 | 2018-07-05 11:01:08.857-05
>>>
>>> perl-5.16.x-all| perl-5.16.x-all| 2015-09-11 13:25:15.002198-05 |
>>> 2015-09-11 13:29:21.361-05
>>>
>>> ovirt-x86_64-stable-6| ovirt-x86_64-stable-6| 2015-09-14 13:30:55.172-05|
>>>
>>> ovirt-x86_64-stable-6-prod | ovirt-x86_64-stable-6-prod | 2015-09-14
>>> 13:48:06.637063-05 |
>>>
>>> other6-x86_64-all| other6-x86_64-all| 2015-07-28 09:20:38.156104-05 |
>>>
>>> epel5-x86_64-all | epel5-x86_64-all | 2016-10-04 18:20:44.846312-05 |
>>> 2017-04-17 12:57:36.859-05
>>>
>>> passenger6-x86_64-prod | passenger6-x86_64-prod | 2016-04-22
>>> 14:35:45.395518-05 |
>>>
>>> perl-5.16.x-nonprod| perl-5.16.x-nonprod| 2015-09-11 13:27:32.261063-05 |
>>>
>>> perl-5.16.x-prod | perl-5.16.x-prod | 2015-09-11 13:26:40.584715-05 |
>>> 2015-09-11 13:29:38.537-05
>>>
>>> other6-x86_64-nonprod| other6-x86_64-nonprod| 2015-07-23
>>> 15:00:03.733479-05 |
>>>
>>> other6-x86_64-prod | other6-x86_64-prod | 2015-07-21 15:10:48.719528-05 |
>>>
>>> epel5-x86_64-prod| epel5-x86_64-prod| 2016-10-04 18:25:38.655383-05 |
>>>
>>> passenger6-x86_64-all| passenger6-x86_64-all| 2016-04-20
>>> 11:37:19.002493-05 | 2016-04-20 11:58:42.312-05
>>>
>>> docker7-x86_64-prod| docker7-x86_64-prod| 2017-08-03 11:42:08.474496-05 |
>>>
>>> centos5-x86_64-nonprod | centos5-x86_64-nonprod | 2015-06-22
>>> 16:16:17.372799-05 |
>>>
>>> other7-x86_64-nonprod| other7-x86_64-nonprod| 2016-07-14
>>> 13:03:10.320136-05 |
>>>
>>> mongo3.2-centos6-x86_64-all| mongo3.2-centos6-x86_64-all| 2016-08-22
>>> 12:21:40.722182-05 | 2018-07-01 12:27:03.019-05
>>>
>>> centos5-x86_64-prod| centos5-x86_64-prod| 2015-06-22 16:20:41.474486-05 |
>>>
>>> passenger6-x86_64-nonprod| passenger6-x86_64-nonprod| 2016-04-20
>>> 12:29:24.677227-05 |
>>>
>>> other7-x86_64-prod | other7-x86_64-prod | 2016-07-14 13:03:47.284295-05 |
>>>
>>> cloudera5.7-x86_64-nonprod | cloudera5.7-x86_64-nonprod | 2016-05-09
>>> 12:10:16.496626-05 | 2016-06-20 13:11:20.62-05
>>>
>>> epel5-x86_64-nonprod | epel5-x86_64-nonprod | 2016-10-04
>>> 18:25:09.844486-05 |
>>>
>>> epel6-x86_64-prod| epel6-x86_64-prod| 2016-03-18 11:52:45.9199-05 |
>>> 2016-08-23 05:07:37.967-05
>>>
>>> spacewalk6-client-all| spacewalk6-client-all| 2017-05-02
>>> 20:53:38.867018-05 | 2018-07-01 22:02:11.386-05
>>>
>>> docker7-x86_64-nonprod | docker7-x86_64-nonprod | 2017-04-07
>>> 15:13:44.158973-05 |
>>>
>>> mongo3.2-centos6-x86_64-nonprod| mongo3.2-centos6-x86_64-nonprod|
>>> 2016-08-22 12:34:18.095059-05 |
>>>
>>> mongo3.2-centos6-x86_64-prod | mongo3.2-centos6-x86_64-prod |
>>> 2016-08-22 12:42:19.161165-05 |
>>>
>>> local6-x86_64-all| local6-x86_64-all| 2015-09-30 08:55:37.657412-05 |
>>> 2016-04-19 07:00:23.632-05
>>>
>>> centos5-x86_64-all | centos5-x86_64-all | 2015-06-22
>>> 15:20:22.085465-05 | 2017-04-17 13:09:39.635-05
>>>
>>> spacewalk5-client-nonprod| spacewalk5-client-nonprod| 2017-05-02
>>> 20:53:20.430795-05 |
>>>
>>> spacewalk5-client-prod | spacewalk5-client-prod | 2017-05-02
>>> 20:53:28.980968-05 |
>>>
>>> spacewalk5-client-all| spacewalk5-client-all| 2017-05-02
>>> 20:53:08.276664-05 | 2018-07-05 10:10:11.665-05
>>>
>>> spacewalk7-client-prod | spacewalk7-client-prod | 2017-05-02
>>> 20:54:32.321635-05 | 2018-07-05 11:01:14.499-05
>>>
>>> epel6-x86_64-nonprod | epel6-x86_64-nonprod | 2016-03-18
>>> 11:52:14.915108-05 | 2018-07-05 10:10:08.774-05
>>>
>>> centos7-x86_64-prod| centos7-x86_64-prod| 2015-11-11 09:02:06.69758-06|
>>>
>>> puppetlabs6-x86_64-prod| puppetlabs6-x86_64-prod| 2016-04-22
>>> 13:46:22.233841-05 | 2018-07-01 13:30:47.635-05
>>>
>>> puppetlabs5-x86_64-nonprod | puppetlabs5-x86_64-nonprod | 2018-03-26
>>> 15:21:59.007749-05 | 2018-07-01 13:00:03.401-05
>>>
>>> puppetlabs5-x86_64-prod| puppetlabs5-x86_64-prod| 2018-03-26
>>> 15:24:23.86552-05| 2018-07-01 13:30:39.025-05
>>>
>>> puppetlabs5-x86_64-all | puppetlabs5-x86_64-all | 2018-03-26
>>> 15:19:04.647981-05 | 2018-07-01 13:31:25.065-05
>>>
>>> other5-x86_64-all| other5-x86_64-all| 2015-08-10 14:16:01.092867-05 |
>>>
>>> other5-x86_64-nonprod| other5-x86_64-nonprod| 2015-08-10
>>> 14:18:05.114541-05 |
>>>
>>> other5-x86_64-prod | other5-x86_64-prod | 2015-08-10 14:19:03.728982-05 |
>>>
>>> centos6-x86_64-nonprod | centos6-x86_64-nonprod | 2015-06-22
>>> 16:24:07.137207-05 |
>>>
>>> centos6-x86_64-prod| centos6-x86_64-prod| 2015-06-22 16:28:51.324002-05 |
>>>
>>> extras7-x86_64-all | extras7-x86_64-all | 2017-08-16 09:13:26.8122-05
>>> | 2018-07-05 10:05:10.626-05
>>>
>>> centos6-x86_64-gitlab-ce-nonprod | centos6-x86_64-gitlab-ce-nonprod |
>>> 2017-04-17 11:43:36.609036-05 | 2018-07-05 10:04:57.277-05
>>>
>>> spacewalk7-server-all| spacewalk7-server-all| 2017-03-28
>>> 15:22:31.851414-05 | 2018-07-05 11:11:31.564-05
>>>
>>> local5-x86_64-all| local5-x86_64-all| 2016-02-24 12:19:36.791459-06 |
>>>
>>> local5-x86_64-nonprod| local5-x86_64-nonprod| 2016-02-24
>>> 12:20:19.404008-06 |
>>>
>>> local5-x86_64-prod | local5-x86_64-prod | 2016-02-24 12:20:45.098532-06 |
>>>
>>> local6-x86_64-nonprod| local6-x86_64-nonprod| 2016-08-22
>>> 20:49:56.7376-05 |
>>>
>>> local7-x86_64-all| local7-x86_64-all| 2016-07-14 13:00:32.511851-05 |
>>>
>>> local7-x86_64-nonprod| local7-x86_64-nonprod| 2016-07-14
>>> 13:02:06.932169-05 |
>>>
>>> local7-x86_64-prod | local7-x86_64-prod | 2016-07-14 13:02:38.496912-05 |
>>>
>>> puppetlabs6-x86_64-all | puppetlabs6-x86_64-all | 2016-04-20
>>> 08:27:56.026914-05 | 2018-07-01 13:30:36.771-05
>>>
>>> spacewalk7-client-nonprod| spacewalk7-client-nonprod| 2017-05-02
>>> 20:54:22.659512-05 | 2018-07-05 11:10:25.009-05
>>>
>>> docker7-x86_64-all | docker7-x86_64-all | 2017-03-22
>>> 12:50:15.332561-05 | 2018-07-05 13:00:02.988-05
>>>
>>> spacewalk7-client-all| spacewalk7-client-all| 2017-05-02
>>> 20:54:13.5076-05 | 2018-07-05 10:04:59.748-05
>>>
>>> local6-x86_64-prod | local6-x86_64-prod | 2015-09-30 08:59:12.679727-05 |
>>>
>>> centos6-x86_64-gitlab-ee-nonprod | centos6-x86_64-gitlab-ee-nonprod |
>>> 2016-04-14 11:39:01.432444-05 | 2018-07-05 11:12:20.525-05
>>>
>>> mysqltools6-x86_64-all | mysqltools6-x86_64-all | 2016-03-17
>>> 12:41:37.44854-05| 2018-07-05 12:00:02.319-05
>>>
>>> mysqltools6-x86_64-nonprod | mysqltools6-x86_64-nonprod | 2016-03-17
>>> 12:58:35.036373-05 |
>>>
>>> mysqltools6-x86_64-prod| mysqltools6-x86_64-prod| 2016-03-17
>>> 12:59:10.969162-05 |
>>>
>>> spacewalk7-server-nonprod| spacewalk7-server-nonprod| 2017-03-28
>>> 15:23:02.210349-05 | 2018-07-05 11:12:47.471-05
>>>
>>> spacewalk7-server-prod | spacewalk7-server-prod | 2017-03-28
>>> 15:23:29.309042-05 | 2017-05-02 20:56:45.247-05
>>>
>>> epel7-x86_64-prod| epel7-x86_64-prod| 2016-03-22 09:48:38.060213-05 |
>>> 2018-07-05 09:57:25.861-05
>>>
>>> puppetlabs6-x86_64-nonprod | puppetlabs6-x86_64-nonprod | 2016-04-20
>>> 12:28:55.337125-05 | 2018-07-01 13:30:43.362-05
>>>
>>> newrelic-noarch-nover| newrelic-noarch-nover| 2016-10-13
>>> 13:54:38.621333-05 | 2016-10-13 14:09:41.778-05
>>>
>>> other7-x86_64-all| other7-x86_64-all| 2016-07-14 13:01:25.848215-05 |
>>> 2018-07-05 14:00:03.714-05
>>>
>>> spacewalk6-client-nonprod| spacewalk6-client-nonprod| 2017-05-02
>>> 20:53:50.507298-05 |
>>>
>>> spacewalk6-client-prod | spacewalk6-client-prod | 2017-05-02
>>> 20:54:00.685324-05 |
>>>
>>> spacewalk6-server-all| spacewalk6-server-all| 2018-06-22
>>> 23:11:30.637054-05 | 2018-07-05 11:01:11.543-05
>>>
>>> puppetlabs7-x86_64-prod| puppetlabs7-x86_64-prod| 2016-07-14
>>> 13:29:04.67033-05| 2018-07-01 13:31:29.425-05
>>>
>>> spacewalk6-server-nonprod| spacewalk6-server-nonprod| 2018-06-22
>>> 23:17:20.660409-05 |
>>>
>>> spacewalk6-server-prod | spacewalk6-server-prod | 2018-06-22
>>> 23:18:02.738869-05 |
>>>
>>> puppetlabs7-x86_64-nonprod | puppetlabs7-x86_64-nonprod | 2016-07-14
>>> 13:28:34.475051-05 | 2018-07-01 13:16:25.948-05
>>>
>>> epel6-x86_64-all | epel6-x86_64-all | 2016-03-18 11:50:17.587171-05 |
>>> 2018-07-05 11:07:42.644-05
>>>
>>> centos6-x86_64-gitlab-ee | centos6-x86_64-gitlab-ee | 2015-12-24
>>> 13:21:10.493684-06 | 2018-07-05 11:08:30.039-05
>>>
>>> puppetlabs7-x86_64-all | puppetlabs7-x86_64-all | 2016-07-14
>>> 12:54:59.388232-05 | 2018-07-01 13:32:02.745-05
>>>
>>> epel7-x86_64-nonprod | epel7-x86_64-nonprod | 2016-03-22
>>> 09:47:34.668867-05 | 2017-04-21 11:08:24.573-05
>>>
>>> centos6-x86_64-all | centos6-x86_64-all | 2015-06-22
>>> 15:19:13.053429-05 | 2018-07-02 01:12:57.768-05
>>>
>>> epel7-x86_64-all | epel7-x86_64-all | 2016-03-22 09:44:48.748142-05 |
>>> 2018-07-05 09:11:28.553-05
>>>
>>> centos7-x86_64-nonprod | centos7-x86_64-nonprod | 2015-10-21
>>> 22:02:28.107902-05 |
>>>
>>> (85 rows)
>>>
>>>
>>> On Thu, Jul 5, 2018 at 11:48 AM Gerald Vogt <vogt at spamcop.net
>>> <mailto:vogt at spamcop.net>> wrote:
>>>
>>> On 05.07.18 16:05, Matt Moldvan wrote:
>>> > How is the server utilization with respect to disk I/O (something
>>> like
>>> > iotop or htop might help here)? Maybe there is something else
>>> blocking
>>>
>>> My server is basically idle. 99% idle, little disk i/o. It doesn't
>>> do anything really.
>>>
>>> > and the server doesn't have enough resources to complete. Have
>>> you
>>> > tried running an strace against the running process?
>>>
>>> If it doesn't have enough resources shouldn't there be an exception?
>>>
>>> For me, it looks more like something doesn't make it into the
>>> database and thus into the persistent state. For instance, I now
>>> have the repodata task at "RUNNING" for three days:
>>>
>>> Channel Repodata: 2018-07-02 08:13:10 CEST RUNNING
>>>
>>> The log file shows this regarding repodata:
>>>
>>> > # fgrep -i repodata rhn_taskomatic_daemon.log
>>> > INFO | jvm 1 | 2018/07/02 08:13:10 | 2018-07-02 08:13:10,584
>>> [Thread-12] INFO com.redhat.rhn.taskomatic.TaskoQuartzHelper - Job
>>> single-channel-repodata-bunch-0 scheduled succesfully.
>>> > INFO | jvm 1 | 2018/07/02 08:13:10 | 2018-07-02 08:13:10,636
>>> [DefaultQuartzScheduler_Worker-8] INFO
>>> com.redhat.rhn.taskomatic.TaskoJob -
>>> single-channel-repodata-bunch-0: bunch channel-repodata-bunch STARTED
>>> > INFO | jvm 1 | 2018/07/02 08:13:10 | 2018-07-02 08:13:10,651
>>> [DefaultQuartzScheduler_Worker-8] DEBUG
>>> com.redhat.rhn.taskomatic.TaskoJob -
>>> single-channel-repodata-bunch-0: task channel-repodata started
>>> > INFO | jvm 1 | 2018/07/02 08:13:10 | 2018-07-02 08:13:10,793
>>> [DefaultQuartzScheduler_Worker-8] INFO
>>> com.redhat.rhn.taskomatic.task.ChannelRepodata - In the queue: 4
>>> > INFO | jvm 1 | 2018/07/02 08:13:11 | 2018-07-02 08:13:11,102
>>> [DefaultQuartzScheduler_Worker-8] DEBUG
>>> com.redhat.rhn.taskomatic.TaskoJob - channel-repodata
>>> (single-channel-repodata-bunch-0) ... running
>>> > INFO | jvm 1 | 2018/07/02 08:13:11 | 2018-07-02 08:13:11,103
>>> [DefaultQuartzScheduler_Worker-8] INFO
>>> com.redhat.rhn.taskomatic.TaskoJob -
>>> single-channel-repodata-bunch-0: bunch channel-repodata-bunch
>>> FINISHED
>>>
>>> So according to the logs the repodata bunch has finished. According
>>> to the web interface it has not. Nothing has been updated in
>>> /var/cache/rhn/repodata/ either. In addition, those four channels
>>> which were still updated haven't been updated either now.
>>>
>>> Thanks,
>>>
>>> Gerald
>>>
>>>
>>>
>>> >
>>> > I also had an (well, many) issue(s) with our Spacewalk server
>>> before
>>> > disabling snapshots in /etc/rhn/rhn.conf. I also increased the
>>> number
>>> > of workers and max repodata work items:
>>> >
>>> > # system snapshots enabled
>>> > enable_snapshots = 0
>>> > ...
>>> > taskomatic.maxmemory=6144
>>> > taskomatic.errata_cache_max_work_items = 500
>>> > taskomatic.channel_repodata_max_work_items = 50
>>> > taskomatic.channel_repodata_workers = 5
>>> >
>>> >
>>> >
>>> > On Thu, Jul 5, 2018 at 4:38 AM Florence Savary
>>> > <florence.savary.fs at gmail.com
>>> <mailto:florence.savary.fs at gmail.com>
>>> <mailto:florence.savary.fs at gmail.com
>>> <mailto:florence.savary.fs at gmail.com>>> wrote:
>>> >
>>> > Hello,
>>> >
>>> > Thanks for sharing your configuration files. They differ very
>>> little
>>> > from mine. I just changed the number of workers in rhn.conf,
>>> but it
>>> > didn't change anything.
>>> >
>>> > I deleted all the channels clones not used by any system and
>>> dating
>>> > back from before May 2018, in order to lower the number of
>>> channels
>>> > in the queue. There were 127 channels in the queue before
>>> these
>>> > deletion (indicated
>>> in /var/log/rhn/rhn_taskomatic_daemon.log), and
>>> > there are 361 of them now ... I must admit I'm confused... I
>>> hoped
>>> > it would reduce the number of channels to process and thus
>>> "help"
>>> > taskomatic, but obviously I was wrong.
>>> >
>>> > I also noticed that the repodata regeneration seems to work
>>> fine for
>>> > existing channels that are not clones, but it is not working
>>> for new
>>> > channels that are not clones (and not working for new
>>> clones but
>>> > nothing new here).
>>> >
>>> > Has anyone got any other idea (even the tiniest) ?
>>> >
>>> > Regards,
>>> > Florence
>>> >
>>> >
>>> > 2018-07-04 15:21 GMT+02:00 Paul Dias - BCX
>>> <paul.dias at bcx.co.za <mailto:paul.dias at bcx.co.za>
>>> > <mailto:paul.dias at bcx.co.za <mailto:paul.dias at bcx.co.za>>>:
>>> >
>>> > Hi,____
>>> >
>>> > __ __
>>> >
>>> > Let me post my settings that I have on my CentOS6 server.
>>> Can’t
>>> > remember but I have one or two others, but his is from
>>> the top
>>> > of my head.____
>>> >
>>> > __ __
>>> >
>>> > /etc/rhn/rhn.conf____
>>> >
>>> > # Added by paul dias increase number of taskomatic workers
>>> > 20180620____
>>> >
>>> > taskomatic.channel_repodata_workers = 3____
>>> >
>>> > taskomatic.java.maxmemory=4096____
>>> >
>>> > __ __
>>> >
>>> > /etc/sysconfig/tomcat6____
>>> >
>>> > JAVA_OPTS="-ea -Xms256m -Xmx512m -Djava.awt.headless=true
>>> > -Dorg.xml.sax.driver=org.apache.xerces.parsers.SAXParser
>>> > -Dorg.apache.tomcat.util.http.Parameters.MAX_COUNT=1024
>>> > -XX:MaxNewSize=256 -XX:-UseConcMarkSweepGC
>>> > -Dnet.sf.ehcache.skipUpdateCheck=true
>>> >
>>> -Djavax.sql.DataSource.Factory=org.apache.commons.dbcp.BasicDataSourceFactory"____
>>>
>>> >
>>> > __ __
>>> >
>>> > /etc/tomcat/server.xml____
>>> >
>>> > <!-- Define an AJP 1.3 Connector on port 8009 -->____
>>> >
>>> > <Connector port="8009" protocol="AJP/1.3"
>>> > redirectPort="8443" URIEncoding="UTF-8"
>>> address="127.0.0.1"
>>> > maxThreads="256" connectionTimeout="20000"/>____
>>> >
>>> > __ __
>>> >
>>> > <Connector port="8009" protocol="AJP/1.3"
>>> > redirectPort="8443" URIEncoding="UTF-8" address="::1"
>>> > maxThreads="256" connectionTimeout="20000"/>____
>>> >
>>> > __ __
>>> >
>>> >
>>> /usr/share/rhn/config-defaults/rhn_taskomatic_daemon.conf____
>>> >
>>> > # Initial Java Heap Size (in MB)____
>>> >
>>> > wrapper.java.initmemory=512____
>>> >
>>> > __ __
>>> >
>>> > # Maximum Java Heap Size (in MB)____
>>> >
>>> > wrapper.java.maxmemory=1512____
>>> >
>>> > # Adjusted by paul 20180620____
>>> >
>>> > __ __
>>> >
>>> > wrapper.ping.timeout=0____
>>> >
>>> > # # adjusted paul dias 20180620____
>>> >
>>> > __ __
>>> >
>>> > __ __
>>> >
>>> > __ __
>>> >
>>> > __ __
>>> >
>>> > __ __
>>> >
>>> > __ __
>>> >
>>> > __ __
>>> >
>>> > __ __
>>> >
>>> > __ __
>>> >
>>> > __ __
>>> >
>>> > __ __
>>> >
>>> > __ __
>>> >
>>> > __ __
>>> >
>>> > __ __
>>> >
>>> > __ __
>>> >
>>> > Regards,____
>>> >
>>> > *Paul Dias____*
>>> >
>>> > Technical Consultant____
>>> >
>>> > 6^th Floor, 8 Boundary Road____
>>> >
>>> > Newlands____
>>> >
>>> > Cape Town____
>>> >
>>> > 7700____
>>> >
>>> > T: +27 (0) 21 681 3149 <tel:+27%2021%20681%203149>
>>> <tel:+27%2021%20681%203149>____
>>> >
>>> > *Meet your future today.____*
>>> >
>>> > *__ __*
>>> >
>>> > __BCX______
>>> >
>>> > __ __
>>> >
>>> > __ __
>>> >
>>> > __ __
>>> >
>>> > __Social-facebook
>>> > <https://www.facebook.com/BCXworld>____Social-twitter
>>> > <https://twitter.com/BCXworld>____Social-linkdin
>>> > <https://za.linkedin.com/BCX>____Social-youtube
>>> > <https://www.youtube.com/BCXworld>______
>>> >
>>> > __ __
>>> >
>>> > __ __
>>> >
>>> > This e-mail is subject to the BCX electronic
>>> communication legal
>>> > notice, available at:
>>> > https://www.bcx.co.za/disclaimers____
>>> >
>>> > /__ __/
>>> >
>>> > /__ __/
>>> >
>>> > __ __
>>> >
>>> > *From:*Paul Dias - BCX
>>> > *Sent:* 02 July 2018 06:53 PM
>>> >
>>> >
>>> > *To:* spacewalk-list at redhat.com
>>> <mailto:spacewalk-list at redhat.com> <mailto:spacewalk-list at redhat.com
>>> <mailto:spacewalk-list at redhat.com>>
>>> > *Subject:* Re: [Spacewalk-list] Taskomatic runs
>>> indefinitely
>>> > without ever generating repodata____
>>> >
>>> > __ __
>>> >
>>> > What I have noticed, if you use
>>> > "spacecmd softchannel_generateyumcache <channel name>"
>>> and then
>>> > go to tasks and run single repodata bunch, you will see
>>> it will
>>> > actually start and generate your channel cache for you
>>> on the
>>> > channel you used the spacecmd on, this works every
>>> time.____
>>> >
>>> > __ __
>>> >
>>> > But yes the task logs just show repodata bunch running
>>> forever.____
>>> >
>>> > __ __
>>> >
>>> > Regards,____
>>> >
>>> > *Paul Dias*____
>>> >
>>> > 6^th Floor, 8 Boundary Road____
>>> >
>>> > Newlands____
>>> >
>>> > Cape Town____
>>> >
>>> > 7700____
>>> >
>>> > T: +27 (0) 21 681 3149 <tel:+27%2021%20681%203149>
>>> <tel:+27%2021%20681%203149>____
>>> >
>>> > __ __
>>> >
>>> > *Meet your future today.*____
>>> >
>>> > **____
>>> >
>>> > BCX____
>>> >
>>> > __ __
>>> >
>>> >
>>> ------------------------------------------------------------------------
>>> >
>>> > *From:*Gerald Vogt <vogt at spamcop.net
>>> <mailto:vogt at spamcop.net> <mailto:vogt at spamcop.net
>>> <mailto:vogt at spamcop.net>>>
>>> > *Sent:* Monday, 02 July 2018 9:45 AM
>>> > *To:* spacewalk-list at redhat.com
>>> <mailto:spacewalk-list at redhat.com> <mailto:spacewalk-list at redhat.com
>>> <mailto:spacewalk-list at redhat.com>>
>>> > *Subject:* Re: [Spacewalk-list] Taskomatic runs
>>> indefinitely
>>> > without ever generating repodata____
>>> >
>>> > ____
>>> >
>>> > After letting the upgraded server sit for a while it
>>> seems only
>>> > a few of
>>> > the task schedules actually finish. By now, only those
>>> tasks
>>> > show up in
>>> > in the task engine status page:
>>> >
>>> > Changelog Cleanup: 2018-07-01 23:00:00 CEST
>>> FINISHED
>>> > Clean Log History: 2018-07-01 23:00:00 CEST
>>> FINISHED
>>> > Compare Config Files: 2018-07-01 23:00:00 CEST
>>> FINISHED
>>> > Daily Summary Mail: 2018-07-01 23:00:00 CEST
>>> FINISHED
>>> > Daily Summary Queue: 2018-07-01 23:00:00 CEST
>>> FINISHED
>>> >
>>> > All the other tasks have disappeared from the list by now.
>>> >
>>> > The repo-sync tasks seem to work. New packages appear
>>> in the
>>> > channel.
>>> > However, the repo build is not running or better it seems
>>> to never
>>> > properly finish.
>>> >
>>> > If I start it manually, it seems to do its work:
>>> >
>>> > > INFO | jvm 1 | 2018/07/02 08:13:10 | 2018-07-02
>>> 08:13:10,584 [Thread-12] INFO
>>> com.redhat.rhn.taskomatic.TaskoQuartzHelper - Job
>>> single-channel-repodata-bunch-0 scheduled succesfully.
>>> > > INFO | jvm 1 | 2018/07/02 08:13:10 | 2018-07-02
>>> 08:13:10,636 [DefaultQuartzScheduler_Worker-8] INFO
>>> com.redhat.rhn.taskomatic.TaskoJob -
>>> single-channel-repodata-bunch-0: bunch channel-repodata-bunch STARTED
>>> > > INFO | jvm 1 | 2018/07/02 08:13:10 | 2018-07-02
>>> 08:13:10,651 [DefaultQuartzScheduler_Worker-8] DEBUG
>>> com.redhat.rhn.taskomatic.TaskoJob -
>>> single-channel-repodata-bunch-0: task channel-repodata started
>>> > > INFO | jvm 1 | 2018/07/02 08:13:10 | 2018-07-02
>>> 08:13:10,793 [DefaultQuartzScheduler_Worker-8] INFO
>>> com.redhat.rhn.taskomatic.task.ChannelRepodata - In the queue: 4
>>> > > INFO | jvm 1 | 2018/07/02 08:13:11 | 2018-07-02
>>> 08:13:11,102 [DefaultQuartzScheduler_Worker-8] DEBUG
>>> com.redhat.rhn.taskomatic.TaskoJob - channel-repodata
>>> (single-channel-repodata-bunch-0) ... running
>>> > > INFO | jvm 1 | 2018/07/02 08:13:11 | 2018-07-02
>>> 08:13:11,103 [DefaultQuartzScheduler_Worker-8] INFO
>>> com.redhat.rhn.taskomatic.TaskoJob -
>>> single-channel-repodata-bunch-0: bunch channel-repodata-bunch
>>> FINISHED
>>> > > INFO | jvm 1 | 2018/07/02 08:13:11 | 2018-07-02
>>> 08:13:11,137 [Thread-677] INFO
>>> com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter - File
>>> Modified Date:2018-06-23 03:48:50 CEST
>>> > > INFO | jvm 1 | 2018/07/02 08:13:11 | 2018-07-02
>>> 08:13:11,137 [Thread-677] INFO
>>> com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter - Channel
>>> Modified Date:2018-07-02 03:45:39 CEST
>>> > > INFO | jvm 1 | 2018/07/02 08:13:11 | 2018-07-02
>>> 08:13:11,211 [Thread-678] INFO
>>> com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter - File
>>> Modified Date:2018-06-23 04:09:51 CEST
>>> > > INFO | jvm 1 | 2018/07/02 08:13:11 | 2018-07-02
>>> 08:13:11,213 [Thread-678] INFO
>>> com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter - Channel
>>> Modified Date:2018-07-02 03:47:55 CEST
>>> > > INFO | jvm 1 | 2018/07/02 08:13:19 | 2018-07-02
>>> 08:13:19,062 [Thread-677] INFO
>>> com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter - Generating
>>> new repository metadata for channel 'epel6-centos6-x86_64'(sha1)
>>> 14401 packages, 11613 errata
>>> > > INFO | jvm 1 | 2018/07/02 08:13:21 | 2018-07-02
>>> 08:13:21,193 [Thread-678] INFO
>>> com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter - Generating
>>> new repository metadata for channel 'epel7-centos7-x86_64'(sha1)
>>> 16282 packages, 10176 errata
>>> > > INFO | jvm 1 | 2018/07/02 08:40:12 | 2018-07-02
>>> 08:40:12,351 [Thread-677] INFO
>>> com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter - Repository
>>> metadata generation for 'epel6-centos6-x86_64' finished in 1613
>>> seconds
>>> > > INFO | jvm 1 | 2018/07/02 08:40:12 | 2018-07-02
>>> 08:40:12,457 [Thread-677] INFO
>>> com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter - File
>>> Modified Date:2018-06-19 06:28:57 CEST
>>> > > INFO | jvm 1 | 2018/07/02 08:40:12 | 2018-07-02
>>> 08:40:12,457 [Thread-677] INFO
>>> com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter - Channel
>>> Modified Date:2018-07-02 04:30:05 CEST
>>> > > INFO | jvm 1 | 2018/07/02 08:40:12 | 2018-07-02
>>> 08:40:12,691 [Thread-677] INFO
>>> com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter - Generating
>>> new repository metadata for channel
>>> 'postgresql96-centos7-x86_64'(sha256) 1032 packages, 0 errata
>>> > > INFO | jvm 1 | 2018/07/02 08:41:51 | 2018-07-02
>>> 08:41:51,710 [Thread-677] INFO
>>> com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter - Repository
>>> metadata generation for 'postgresql96-centos7-x86_64' finished in 98
>>> seconds
>>> > > INFO | jvm 1 | 2018/07/02 08:41:51 | 2018-07-02
>>> 08:41:51,803 [Thread-677] INFO
>>> com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter - File
>>> Modified Date:2018-06-20 05:08:38 CEST
>>> > > INFO | jvm 1 | 2018/07/02 08:41:51 | 2018-07-02
>>> 08:41:51,803 [Thread-677] INFO
>>> com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter - Channel
>>> Modified Date:2018-07-02 04:00:00 CEST
>>> > > INFO | jvm 1 | 2018/07/02 08:41:51 | 2018-07-02
>>> 08:41:51,923 [Thread-677] INFO
>>> com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter - Generating
>>> new repository metadata for channel
>>> 'postgresql10-centos6-x86_64'(sha512) 436 packages, 0 errata
>>> > > INFO | jvm 1 | 2018/07/02 08:42:26 | 2018-07-02
>>> 08:42:26,479 [Thread-677] INFO
>>> com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter - Repository
>>> metadata generation for 'postgresql10-centos6-x86_64' finished in 34
>>> seconds
>>> > > INFO | jvm 1 | 2018/07/02 08:45:01 | 2018-07-02
>>> 08:45:01,697 [Thread-678] INFO
>>> com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter - Repository
>>> metadata generation for 'epel7-centos7-x86_64' finished in 1900
>>> seconds
>>> >
>>> > yet, the task remains in RUNNING. And for whatever
>>> reason it
>>> > only seems
>>> > to work some channels. I find a total of 20 repos syncing
>>> in the
>>> > logs of
>>> > the updated server compared to 42 repos syncing in the
>>> logs of
>>> > the old.
>>> > I don't really see the difference between those 20 repos
>>> syncing
>>> > and
>>> > those other 22 not. First I suspected channels with
>>> custom quartz
>>> > schedules, but then I found channels in both groups.
>>> >
>>> > So I don't know how to troubleshoot this any further. The
>>> > repodata task
>>> > which I have started 1,5 hours ago is still at
>>> "RUNNING". The
>>> > channels
>>> > for which the sync works have been updated. I don't know
>>> why it
>>> > is still
>>> > running. Server load is back down...
>>> >
>>> > Thanks,
>>> >
>>> > Gerald
>>> >
>>> > On 22.06.18 19:12, Gerald Vogt wrote:
>>> > > I have the same problem after upgrading from 2.6 to 2.8
>>> on CentOS 6.9. I
>>> > > have even increased the memory as suggested by that
>>> link but it makes no
>>> > > differences. None of the scheduled tasks are running. I
>>> can run a bunch
>>> > > manually. But the scheduler doesn't seem to work. Last
>>> execution times
>>> > > on the task engine status pages are still at timestamps
>>> from before the
>>> > > upgrade. -Gerald
>>> > >
>>> > >
>>> > >
>>> > > On 22.06.18 14:15, Avi Miller wrote:
>>> > >> Hi,
>>> > >>
>>> > >>> On 22 Jun 2018, at 5:51 pm, Florence Savary
>>> > >>> <florence.savary.fs at gmail.com
>>> <mailto:florence.savary.fs at gmail.com>
>>> > <mailto:florence.savary.fs at gmail.com
>>> <mailto:florence.savary.fs at gmail.com>>> wrote:
>>> > >>>
>>> > >>> When using taskotop, we can see a line for the
>>> channel-repodata task,
>>> > >>> we see it is running, but there is never any channel
>>> displayed in the
>>> > >>> Channel column. We can also see the task marked as
>>> running in the
>>> > >>> Admin tab of the WebUI, but if we let it, it never
>>> stops. The task
>>> > >>> runs indefinitely, whithout ever doing anything.
>>> > >>
>>> > >> If you've never modified the default memory settings,
>>> Taskomatic is
>>> > >> probably running out of memory and task is crashing.
>>> This is a known
>>> > >> issue, particularly when you sync large repos.
>>> > >>
>>> > >> I would suggest increasing the memory assigned to
>>> Taskomatic to see if
>>> > >> that resolves the issue. You will need to restart it
>>> after making
>>> > >> these changes:
>>> > >>
>>>
>>> https://docs.oracle.com/cd/E92593_01/E90695/html/swk24-issues-memory.html
>>> > >>
>>> > >> Cheers,
>>> > >> Avi
>>> > >>
>>> > >> --
>>> > >> Oracle <http://www.oracle.com>
>>> > >> Avi Miller | Product Management Director | +61 (3)
>>> 8616 3496 <tel:+61%203%208616%203496> <tel:+61%203%208616%203496>
>>> > >> Oracle Linux and Virtualization
>>> > >> 417 St Kilda Road, Melbourne, Victoria 3004 Australia
>>> > >>
>>> > >>
>>> > >> _______________________________________________
>>> > >> Spacewalk-list mailing list
>>> > >> Spacewalk-list at redhat.com
>>> <mailto:Spacewalk-list at redhat.com> <mailto:Spacewalk-list at redhat.com
>>> <mailto:Spacewalk-list at redhat.com>>
>>> > >> https://www.redhat.com/mailman/listinfo/spacewalk-list
>>> > >>
>>> > >
>>> > > _______________________________________________
>>> > > Spacewalk-list mailing list
>>> > > Spacewalk-list at redhat.com
>>> <mailto:Spacewalk-list at redhat.com> <mailto:Spacewalk-list at redhat.com
>>> <mailto:Spacewalk-list at redhat.com>>
>>> > > https://www.redhat.com/mailman/listinfo/spacewalk-list
>>> >
>>> > ____
>>> >
>>> >
>>> > _______________________________________________
>>> > Spacewalk-list mailing list
>>> > Spacewalk-list at redhat.com <mailto:Spacewalk-list at redhat.com>
>>> <mailto:Spacewalk-list at redhat.com <mailto:Spacewalk-list at redhat.com>>
>>> > https://www.redhat.com/mailman/listinfo/spacewalk-list
>>> >
>>> >
>>> > _______________________________________________
>>> > Spacewalk-list mailing list
>>> > Spacewalk-list at redhat.com <mailto:Spacewalk-list at redhat.com>
>>> <mailto:Spacewalk-list at redhat.com <mailto:Spacewalk-list at redhat.com>>
>>> > https://www.redhat.com/mailman/listinfo/spacewalk-list
>>> >
>>> >
>>> >
>>> > _______________________________________________
>>> > Spacewalk-list mailing list
>>> > Spacewalk-list at redhat.com <mailto:Spacewalk-list at redhat.com>
>>> > https://www.redhat.com/mailman/listinfo/spacewalk-list
>>> >
>>>
>>> _______________________________________________
>>> Spacewalk-list mailing list
>>> Spacewalk-list at redhat.com <mailto:Spacewalk-list at redhat.com>
>>> https://www.redhat.com/mailman/listinfo/spacewalk-list
>>>
>>>
>>>
>>> _______________________________________________
>>> Spacewalk-list mailing list
>>> Spacewalk-list at redhat.com
>>> https://www.redhat.com/mailman/listinfo/spacewalk-list
>>>
>>
>> _______________________________________________
>> Spacewalk-list mailing list
>> Spacewalk-list at redhat.com
>> https://www.redhat.com/mailman/listinfo/spacewalk-list
>
>
>
>
> _______________________________________________
> Spacewalk-list mailing list
> Spacewalk-list at redhat.com
> https://www.redhat.com/mailman/listinfo/spacewalk-list
>
More information about the Spacewalk-list
mailing list