[Pulp-list] Missing celery workers
Brian Bouterse
bbouters at redhat.com
Mon Nov 30 16:14:52 UTC 2015
TL;DR: I'm continuing to work on the missing workers on EL/CentOS 6
issue w/ 2.7.0.
I agree that switching to the newer Qpid stack for EL6 will not resolve
the missing workers issue on 2.7. Since I've reproduced it with rabbitMQ
I don't think it's broker related.
Switching to the newly available Qpid stack [0] for EL6 does resolve
issue #1340 (Qpid could not be installed on EL/CentOS 6)
[0]: https://pulp.plan.io/issues/1340#note-4
-Brian
On 11/29/2015 11:48 AM, Joel Golden wrote:
> That is correct. Although behavior was different after the switch and
> restart of services, (i.e. my run sync jobs did not immediately crash),
> I still received the missing workers messages. Unfortunately, the only
> fix I found was to migrated to CentOS 7, which was much less painful
> than debugging 6.
>
> On Sun, Nov 29, 2015 at 9:43 AM, Ashby, Jason (IMS) <AshbyJ at imsweb.com
> <mailto:AshbyJ at imsweb.com>> wrote:
>
> Hmm, I don’t think this would fix the missing worker issue for
> people like myself using rabbitmq instead of qpid. Though it may
> fix the/other/ issue of broken EL6 qpid dependencies.____
>
> __ __
>
> *From:*pulp-list-bounces at redhat.com
> <mailto:pulp-list-bounces at redhat.com>
> [mailto:pulp-list-bounces at redhat.com
> <mailto:pulp-list-bounces at redhat.com>] *On Behalf Of *Joel Golden
> *Sent:* Thursday, November 26, 2015 12:27 AM
>
>
> *To:* pulp-list at redhat.com <mailto:pulp-list at redhat.com>
> *Subject:* Re: [Pulp-list] Missing celery workers____
>
> __ __
>
> Fix - replace qpid-cpp-server:____
>
> __ __
>
> Installing: ____
>
> qpid-cpp-server-linearstore x86_64 0.34-4.el6____
>
> replacing qpid-cpp-server-store.x86_64 0.26-9.el6____
>
> Updating: ____
>
> python-qpid noarch 0.32-12.el6____
>
> python-qpid-common noarch 0.32-12.el6____
>
> python-qpid-qmf x86_64 0.32-1.el6____
>
> qpid-cpp-client x86_64 0.34-4.el6____
>
> qpid-cpp-server x86_64 0.34-4.el6____
>
> qpid-proton-c x86_64 0.10-2.el6____
>
> qpid-qmf x86_64 0.32-1.el6____
>
> Installing for dependencies:____
>
> python-saslwrapper x86_64 0.22-5.el6____
>
> __ __
>
> Enjoy! Joel Golden____
>
> __ __
>
> Due to an existing set of deprecated Qpid packages in RHEL 6 and
> CentOS 6, we cannot at present provide updated Qpid packages
> directly in EPEL 6. We offer an alternative repository at Fedora
> Copr.____
>
> Qpid at Copr ____
>
> Copr repo file for RHEL 6 and CentOS 6____
>
> https://copr.fedoraproject.org/coprs/irina/qpid/repo/epel-6/irina-qpid-epel-6.repo____
>
> Copr Qpid GPG public key____
>
> https://qpid.apache.org/copr-qpid-pubkey.gpg____
>
> __ __
>
> On Thu, Nov 19, 2015 at 5:08 AM, Miller, Jeffrey L
> <jeff-l-miller at uiowa.edu <mailto:jeff-l-miller at uiowa.edu>> wrote:____
>
> In answer to posed questions:
> - RHEL 6 x86_64
> - qpid-cpp-server
> - Every 60 seconds for me as well. /var/log/messages loos as below.
>
> -Jeffrey____
>
>
>
>
>
> -----Original Message-----
> From: pulp-list-bounces at redhat.com
> <mailto:pulp-list-bounces at redhat.com>
> [mailto:pulp-list-bounces at redhat.com
> <mailto:pulp-list-bounces at redhat.com>] On Behalf Of Ashby, Jason
> (IMS)
> Sent: Wednesday, November 18, 2015 10:38 AM
> To: Brian Bouterse <bbouters at redhat.com
> <mailto:bbouters at redhat.com>>; pulp-list at redhat.com
> <mailto:pulp-list at redhat.com>
> Subject: Re: [Pulp-list] Missing celery workers
>
> For me it is consistently every 60 seconds. Here are logs from
> the last 10 minutes. I also set loglevel to DEBUG for each
> config file in /etc/default/pulp* and restarted the services,
> but I'm not seeing any DEBUG stuff in the logs.
>
>
> $ sudo grep missing /var/log/messages
> Nov 18 11:23:34 pulp01 pulp: pulp.server.async.scheduler:ERROR:
> Worker 'reserved_resource_worker-0 at pulp01' has gone missing,
> removing from list of workers Nov 18 11:23:34 pulp01 pulp:
> pulp.server.async.tasks:ERROR: The worker named
> reserved_resource_worker-0 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:23:34 pulp01 pulp: pulp.server.async.scheduler:ERROR:
> Worker 'reserved_resource_worker-2 at pulp01' has gone missing,
> removing from list of workers Nov 18 11:23:34 pulp01 pulp:
> pulp.server.async.tasks:ERROR: The worker named
> reserved_resource_worker-2 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:23:34 pulp01 pulp: pulp.server.async.scheduler:ERROR:
> Worker 'reserved_resource_worker-3 at pulp01' has gone missing,
> removing from list of workers Nov 18 11:23:34 pulp01 pulp:
> pulp.server.async.tasks:ERROR: The worker named
> reserved_resource_worker-3 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:23:34 pulp01 pulp: pulp.server.async.scheduler:ERROR:
> Worker 'reserved_resource_worker-1 at pulp01' has gone missing,
> removing from list of workers Nov 18 11:23:34 pulp01 pulp:
> pulp.server.async.tasks:ERROR: The worker named
> reserved_resource_worker-1 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:23:34 pulp01 pulp: pulp.server.async.scheduler:ERROR:
> Worker 'resource_manager at pulp01' has gone missing, removing from
> list of workers Nov 18 11:23:34 pulp01 pulp:
> pulp.server.async.tasks:ERROR: The worker named
> resource_manager at pulp01 is missing. Canceling the tasks in its
> queue.
> Nov 18 11:24:34 pulp01 pulp: pulp.server.async.scheduler:ERROR:
> Worker 'reserved_resource_worker-2 at pulp01' has gone missing,
> removing from list of workers Nov 18 11:24:34 pulp01 pulp:
> pulp.server.async.tasks:ERROR: The worker named
> reserved_resource_worker-2 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:24:34 pulp01 pulp: pulp.server.async.scheduler:ERROR:
> Worker 'reserved_resource_worker-0 at pulp01' has gone missing,
> removing from list of workers Nov 18 11:24:34 pulp01 pulp:
> pulp.server.async.tasks:ERROR: The worker named
> reserved_resource_worker-0 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:24:34 pulp01 pulp: pulp.server.async.scheduler:ERROR:
> Worker 'reserved_resource_worker-3 at pulp01' has gone missing,
> removing from list of workers Nov 18 11:24:34 pulp01 pulp:
> pulp.server.async.tasks:ERROR: The worker named
> reserved_resource_worker-3 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:24:34 pulp01 pulp: pulp.server.async.scheduler:ERROR:
> Worker 'reserved_resource_worker-1 at pulp01' has gone missing,
> removing from list of workers Nov 18 11:24:34 pulp01 pulp:
> pulp.server.async.tasks:ERROR: The worker named
> reserved_resource_worker-1 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:24:34 pulp01 pulp: pulp.server.async.scheduler:ERROR:
> Worker 'resource_manager at pulp01' has gone missing, removing from
> list of workers Nov 18 11:24:34 pulp01 pulp:
> pulp.server.async.tasks:ERROR: The worker named
> resource_manager at pulp01 is missing. Canceling the tasks in its
> queue.
> Nov 18 11:25:34 pulp01 pulp: pulp.server.async.scheduler:ERROR:
> Worker 'resource_manager at pulp01' has gone missing, removing from
> list of workers Nov 18 11:25:34 pulp01 pulp:
> pulp.server.async.tasks:ERROR: The worker named
> resource_manager at pulp01 is missing. Canceling the tasks in its
> queue.
> Nov 18 11:25:34 pulp01 pulp: pulp.server.async.scheduler:ERROR:
> Worker 'reserved_resource_worker-0 at pulp01' has gone missing,
> removing from list of workers Nov 18 11:25:34 pulp01 pulp:
> pulp.server.async.tasks:ERROR: The worker named
> reserved_resource_worker-0 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:25:34 pulp01 pulp: pulp.server.async.scheduler:ERROR:
> Worker 'reserved_resource_worker-1 at pulp01' has gone missing,
> removing from list of workers Nov 18 11:25:34 pulp01 pulp:
> pulp.server.async.tasks:ERROR: The worker named
> reserved_resource_worker-1 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:26:34 pulp01 pulp: pulp.server.async.scheduler:ERROR:
> Worker 'reserved_resource_worker-0 at pulp01' has gone missing,
> removing from list of workers Nov 18 11:26:34 pulp01 pulp:
> pulp.server.async.tasks:ERROR: The worker named
> reserved_resource_worker-0 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:26:34 pulp01 pulp: pulp.server.async.scheduler:ERROR:
> Worker 'reserved_resource_worker-2 at pulp01' has gone missing,
> removing from list of workers Nov 18 11:26:34 pulp01 pulp:
> pulp.server.async.tasks:ERROR: The worker named
> reserved_resource_worker-2 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:26:34 pulp01 pulp: pulp.server.async.scheduler:ERROR:
> Worker 'reserved_resource_worker-1 at pulp01' has gone missing,
> removing from list of workers Nov 18 11:26:34 pulp01 pulp:
> pulp.server.async.tasks:ERROR: The worker named
> reserved_resource_worker-1 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:26:34 pulp01 pulp: pulp.server.async.scheduler:ERROR:
> Worker 'reserved_resource_worker-3 at pulp01' has gone missing,
> removing from list of workers Nov 18 11:26:34 pulp01 pulp:
> pulp.server.async.tasks:ERROR: The worker named
> reserved_resource_worker-3 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:26:34 pulp01 pulp: pulp.server.async.scheduler:ERROR:
> Worker 'resource_manager at pulp01' has gone missing, removing from
> list of workers Nov 18 11:26:34 pulp01 pulp:
> pulp.server.async.tasks:ERROR: The worker named
> resource_manager at pulp01 is missing. Canceling the tasks in its
> queue.
> Nov 18 11:27:34 pulp01 pulp: pulp.server.async.scheduler:ERROR:
> Worker 'reserved_resource_worker-3 at pulp01' has gone missing,
> removing from list of workers Nov 18 11:27:34 pulp01 pulp:
> pulp.server.async.tasks:ERROR: The worker named
> reserved_resource_worker-3 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:27:34 pulp01 pulp: pulp.server.async.scheduler:ERROR:
> Worker 'reserved_resource_worker-0 at pulp01' has gone missing,
> removing from list of workers Nov 18 11:27:34 pulp01 pulp:
> pulp.server.async.tasks:ERROR: The worker named
> reserved_resource_worker-0 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:27:34 pulp01 pulp: pulp.server.async.scheduler:ERROR:
> Worker 'reserved_resource_worker-2 at pulp01' has gone missing,
> removing from list of workers Nov 18 11:27:34 pulp01 pulp:
> pulp.server.async.tasks:ERROR: The worker named
> reserved_resource_worker-2 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:27:34 pulp01 pulp: pulp.server.async.scheduler:ERROR:
> Worker 'reserved_resource_worker-1 at pulp01' has gone missing,
> removing from list of workers Nov 18 11:27:34 pulp01 pulp:
> pulp.server.async.tasks:ERROR: The worker named
> reserved_resource_worker-1 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:27:34 pulp01 pulp: pulp.server.async.scheduler:ERROR:
> Worker 'resource_manager at pulp01' has gone missing, removing from
> list of workers Nov 18 11:27:34 pulp01 pulp:
> pulp.server.async.tasks:ERROR: The worker named
> resource_manager at pulp01 is missing. Canceling the tasks in its
> queue.
> Nov 18 11:28:34 pulp01 pulp: pulp.server.async.scheduler:ERROR:
> Worker 'reserved_resource_worker-3 at pulp01' has gone missing,
> removing from list of workers Nov 18 11:28:34 pulp01 pulp:
> pulp.server.async.tasks:ERROR: The worker named
> reserved_resource_worker-3 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:28:34 pulp01 pulp: pulp.server.async.scheduler:ERROR:
> Worker 'reserved_resource_worker-0 at pulp01' has gone missing,
> removing from list of workers Nov 18 11:28:34 pulp01 pulp:
> pulp.server.async.tasks:ERROR: The worker named
> reserved_resource_worker-0 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:28:34 pulp01 pulp: pulp.server.async.scheduler:ERROR:
> Worker 'reserved_resource_worker-2 at pulp01' has gone missing,
> removing from list of workers Nov 18 11:28:34 pulp01 pulp:
> pulp.server.async.tasks:ERROR: The worker named
> reserved_resource_worker-2 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:28:34 pulp01 pulp: pulp.server.async.scheduler:ERROR:
> Worker 'reserved_resource_worker-1 at pulp01' has gone missing,
> removing from list of workers Nov 18 11:28:34 pulp01 pulp:
> pulp.server.async.tasks:ERROR: The worker named
> reserved_resource_worker-1 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:28:34 pulp01 pulp: pulp.server.async.scheduler:ERROR:
> Worker 'resource_manager at pulp01' has gone missing, removing from
> list of workers Nov 18 11:28:34 pulp01 pulp:
> pulp.server.async.tasks:ERROR: The worker named
> resource_manager at pulp01 is missing. Canceling the tasks in its
> queue.
> Nov 18 11:29:34 pulp01 pulp: pulp.server.async.scheduler:ERROR:
> Worker 'reserved_resource_worker-0 at pulp01' has gone missing,
> removing from list of workers Nov 18 11:29:34 pulp01 pulp:
> pulp.server.async.tasks:ERROR: The worker named
> reserved_resource_worker-0 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:29:34 pulp01 pulp: pulp.server.async.scheduler:ERROR:
> Worker 'reserved_resource_worker-2 at pulp01' has gone missing,
> removing from list of workers Nov 18 11:29:34 pulp01 pulp:
> pulp.server.async.tasks:ERROR: The worker named
> reserved_resource_worker-2 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:29:34 pulp01 pulp: pulp.server.async.scheduler:ERROR:
> Worker 'reserved_resource_worker-1 at pulp01' has gone missing,
> removing from list of workers Nov 18 11:29:34 pulp01 pulp:
> pulp.server.async.tasks:ERROR: The worker named
> reserved_resource_worker-1 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:29:34 pulp01 pulp: pulp.server.async.scheduler:ERROR:
> Worker 'reserved_resource_worker-3 at pulp01' has gone missing,
> removing from list of workers Nov 18 11:29:34 pulp01 pulp:
> pulp.server.async.tasks:ERROR: The worker named
> reserved_resource_worker-3 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:29:34 pulp01 pulp: pulp.server.async.scheduler:ERROR:
> Worker 'resource_manager at pulp01' has gone missing, removing from
> list of workers Nov 18 11:29:34 pulp01 pulp:
> pulp.server.async.tasks:ERROR: The worker named
> resource_manager at pulp01 is missing. Canceling the tasks in its
> queue.
> Nov 18 11:30:34 pulp01 pulp: pulp.server.async.scheduler:ERROR:
> Worker 'reserved_resource_worker-0 at pulp01' has gone missing,
> removing from list of workers Nov 18 11:30:34 pulp01 pulp:
> pulp.server.async.tasks:ERROR: The worker named
> reserved_resource_worker-0 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:30:34 pulp01 pulp: pulp.server.async.scheduler:ERROR:
> Worker 'reserved_resource_worker-2 at pulp01' has gone missing,
> removing from list of workers Nov 18 11:30:34 pulp01 pulp:
> pulp.server.async.tasks:ERROR: The worker named
> reserved_resource_worker-2 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:30:34 pulp01 pulp: pulp.server.async.scheduler:ERROR:
> Worker 'reserved_resource_worker-1 at pulp01' has gone missing,
> removing from list of workers Nov 18 11:30:34 pulp01 pulp:
> pulp.server.async.tasks:ERROR: The worker named
> reserved_resource_worker-1 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:30:34 pulp01 pulp: pulp.server.async.scheduler:ERROR:
> Worker 'reserved_resource_worker-3 at pulp01' has gone missing,
> removing from list of workers Nov 18 11:30:34 pulp01 pulp:
> pulp.server.async.tasks:ERROR: The worker named
> reserved_resource_worker-3 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:30:34 pulp01 pulp: pulp.server.async.scheduler:ERROR:
> Worker 'resource_manager at pulp01' has gone missing, removing from
> list of workers Nov 18 11:30:34 pulp01 pulp:
> pulp.server.async.tasks:ERROR: The worker named
> resource_manager at pulp01 is missing. Canceling the tasks in its
> queue.
> Nov 18 11:31:34 pulp01 pulp: pulp.server.async.scheduler:ERROR:
> Worker 'reserved_resource_worker-0 at pulp01' has gone missing,
> removing from list of workers Nov 18 11:31:34 pulp01 pulp:
> pulp.server.async.tasks:ERROR: The worker named
> reserved_resource_worker-0 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:31:34 pulp01 pulp: pulp.server.async.scheduler:ERROR:
> Worker 'reserved_resource_worker-2 at pulp01' has gone missing,
> removing from list of workers Nov 18 11:31:34 pulp01 pulp:
> pulp.server.async.tasks:ERROR: The worker named
> reserved_resource_worker-2 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:31:34 pulp01 pulp: pulp.server.async.scheduler:ERROR:
> Worker 'reserved_resource_worker-1 at pulp01' has gone missing,
> removing from list of workers Nov 18 11:31:34 pulp01 pulp:
> pulp.server.async.tasks:ERROR: The worker named
> reserved_resource_worker-1 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:31:34 pulp01 pulp: pulp.server.async.scheduler:ERROR:
> Worker 'reserved_resource_worker-3 at pulp01' has gone missing,
> removing from list of workers Nov 18 11:31:34 pulp01 pulp:
> pulp.server.async.tasks:ERROR: The worker named
> reserved_resource_worker-3 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:31:34 pulp01 pulp: pulp.server.async.scheduler:ERROR:
> Worker 'resource_manager at pulp01' has gone missing, removing from
> list of workers Nov 18 11:31:34 pulp01 pulp:
> pulp.server.async.tasks:ERROR: The worker named
> resource_manager at pulp01 is missing. Canceling the tasks in its
> queue.
> Nov 18 11:32:35 pulp01 pulp: pulp.server.async.scheduler:ERROR:
> Worker 'reserved_resource_worker-0 at pulp01' has gone missing,
> removing from list of workers Nov 18 11:32:35 pulp01 pulp:
> pulp.server.async.tasks:ERROR: The worker named
> reserved_resource_worker-0 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:32:35 pulp01 pulp: pulp.server.async.scheduler:ERROR:
> Worker 'reserved_resource_worker-2 at pulp01' has gone missing,
> removing from list of workers Nov 18 11:32:35 pulp01 pulp:
> pulp.server.async.tasks:ERROR: The worker named
> reserved_resource_worker-2 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:32:35 pulp01 pulp: pulp.server.async.scheduler:ERROR:
> Worker 'reserved_resource_worker-1 at pulp01' has gone missing,
> removing from list of workers Nov 18 11:32:35 pulp01 pulp:
> pulp.server.async.tasks:ERROR: The worker named
> reserved_resource_worker-1 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:32:35 pulp01 pulp: pulp.server.async.scheduler:ERROR:
> Worker 'reserved_resource_worker-3 at pulp01' has gone missing,
> removing from list of workers Nov 18 11:32:35 pulp01 pulp:
> pulp.server.async.tasks:ERROR: The worker named
> reserved_resource_worker-3 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:32:35 pulp01 pulp: pulp.server.async.scheduler:ERROR:
> Worker 'resource_manager at pulp01' has gone missing, removing from
> list of workers Nov 18 11:32:35 pulp01 pulp:
> pulp.server.async.tasks:ERROR: The worker named
> resource_manager at pulp01 is missing. Canceling the tasks in its
> queue.
>
> -----Original Message-----
> From: pulp-list-bounces at redhat.com
> <mailto:pulp-list-bounces at redhat.com>
> [mailto:pulp-list-bounces at redhat.com
> <mailto:pulp-list-bounces at redhat.com>] On Behalf Of Brian Bouterse
> Sent: Wednesday, November 18, 2015 11:05 AM
> To: pulp-list at redhat.com <mailto:pulp-list at redhat.com>
> Subject: Re: [Pulp-list] Missing celery workers
>
> Jason and Jeffrey,
>
> Thanks for reporting this. I've written up a bug [0] and I am
> investigating the root cause.
>
> On the bug are you able to leave some answers to these questions?
>
> - Can you confirm that it affects both RabbitMQ and Qpid usage?
> - Can you confirm that the workers "go missing" and then return,
> and then "go missing" in a continuous cycle? I expect it to
> happen every 90 seconds.
>
> - Jeffrey specifically, what OS are you using?
>
> [0]: https://pulp.plan.io/issues/1380
>
> Thanks,
> Brian
>
> On 11/18/2015 09:33 AM, Miller, Jeffrey L wrote:
> > I am seeing this behavior as well after upgrading from 2.6 to 2.7.
> > However, I am using qpid not rabbitmq.
> >
> >
> >
> > -Jeffrey
> >
> >
> >
> >
> >
> >
> >
> > *From:* pulp-list-bounces at redhat.com
> <mailto:pulp-list-bounces at redhat.com>
> > [mailto:pulp-list-bounces at redhat.com
> <mailto:pulp-list-bounces at redhat.com>] *On Behalf Of *Ashby, Jason
> > (IMS)
> > *Sent:* Wednesday, November 18, 2015 8:29 AM
> > *To:* pulp-list at redhat.com <mailto:pulp-list at redhat.com>
> > *Subject:* [Pulp-list] Missing celery workers
> >
> >
> >
> > Hi all,
> >
> > I'm hitting another issue with the upgrade to Pulp 2.7.0 +
> changing
> > from qpid to rabbitmq for messaging. The workers are continuously
> > going missing, every minute or so. The effect is that the
> tasks in
> > the task list stay in a Waiting state and are never completed.
> >
> >
> >
> > Rabbitmq looks healthy; I see successful accepted connections
> per the
> > logs and can see a bunch of connections in the rabbitmq
> management GUI.
> > I'm kind of stuck as far as troubleshooting goes. Any tips on
> what
> > else to investigate?
> >
> >
> >
> > Pulp and rabbitmq servers are both CentOS 6.
> >
> >
> >
> > # /var/log/messages
> >
> > Nov 18 08:53:56 pulp01 pulp: celery.worker.consumer:INFO: missed
> > heartbeat from resource_manager at pulp01
> >
> > Nov 18 09:05:46 pulp01 pulp:
> pulp.server.async.worker_watcher:INFO:
> > New worker 'reserved_resource_worker-3 at pulp01' discovered
> >
> > Nov 18 09:05:46 pulp01 pulp:
> pulp.server.async.worker_watcher:INFO:
> > New worker 'reserved_resource_worker-1 at pulp01' discovered
> >
> > Nov 18 09:05:46 pulp01 pulp:
> pulp.server.async.worker_watcher:INFO:
> > New worker 'reserved_resource_worker-2 at pulp01' discovered
> >
> > Nov 18 09:05:46 pulp01 pulp:
> pulp.server.async.worker_watcher:INFO:
> > New worker 'reserved_resource_worker-0 at pulp01' discovered
> >
> > Nov 18 09:05:56 pulp01 pulp:
> pulp.server.async.worker_watcher:INFO:
> > New worker 'resource_manager at pulp01' discovered
> >
> > Nov 18 09:06:46 pulp01 pulp:
> pulp.server.async.scheduler:ERROR: Worker
> > 'reserved_resource_worker-3 at pulp01' has gone missing, removing
> from
> > list of work
> >
> > ers
> >
> > Nov 18 09:06:46 pulp01 pulp: pulp.server.async.tasks:ERROR:
> The worker
> > named reserved_resource_worker-3 at pulp01 is missing. Canceling the
> > tasks in its q
> >
> > ueue.
> >
> > Nov 18 09:06:46 pulp01 pulp:
> pulp.server.async.scheduler:ERROR: Worker
> > 'reserved_resource_worker-1 at pulp01' has gone missing, removing
> from
> > list of work
> >
> > ers
> >
> > Nov 18 09:06:46 pulp01 pulp: pulp.server.async.tasks:ERROR:
> The worker
> > named reserved_resource_worker-1 at pulp01 is missing. Canceling the
> > tasks in its q
> >
> > ueue.
> >
> > Nov 18 09:06:46 pulp01 pulp:
> pulp.server.async.scheduler:ERROR: Worker
> > 'reserved_resource_worker-2 at pulp01' has gone missing, removing
> from
> > list of work
> >
> > ers
> >
> > Nov 18 09:06:46 pulp01 pulp: pulp.server.async.tasks:ERROR:
> The worker
> > named reserved_resource_worker-2 at pulp01 is missing. Canceling the
> > tasks in its q
> >
> > ueue.
> >
> > Nov 18 09:06:46 pulp01 pulp:
> pulp.server.async.scheduler:ERROR: Worker
> > 'reserved_resource_worker-0 at pulp01' has gone missing, removing
> from
> > list of work
> >
> > ers
> >
> > Nov 18 09:06:46 pulp01 pulp: pulp.server.async.tasks:ERROR:
> The worker
> > named reserved_resource_worker-0 at pulp01 is missing. Canceling the
> > tasks in its q
> >
> > ueue.
> >
> > Nov 18 09:06:46 pulp01 pulp:
> pulp.server.async.scheduler:ERROR: Worker
> > 'resource_manager at pulp01' has gone missing, removing from list of
> > workers
> >
> > Nov 18 09:06:46 pulp01 pulp: pulp.server.async.tasks:ERROR:
> The worker
> > named resource_manager at pulp01 is missing. Canceling the tasks
> in its queue.
> >
> > Nov 18 09:06:46 pulp01 pulp:
> pulp.server.async.scheduler:ERROR: There
> > are 0 pulp_resource_manager processes running. Pulp will not
> operate
> > correctly without
> >
> > at least one pulp_resource_mananger process running.
> >
> >
> >
> >
> ----------------------------------------------------------------------
> > --
> >
> >
> > Information in this e-mail may be confidential. It is intended
> only
> > for the addressee(s) identified above. If you are not the
> > addressee(s), or an employee or agent of the addressee(s),
> please note
> > that any dissemination, distribution, or copying of this
> communication
> > is strictly prohibited. If you have received this e-mail in error,
> > please notify the sender of the error.
> >
> >
> >
> > _______________________________________________
> > Pulp-list mailing list
> > Pulp-list at redhat.com <mailto:Pulp-list at redhat.com>
> > https://www.redhat.com/mailman/listinfo/pulp-list
> >
>
> _______________________________________________
> Pulp-list mailing list
> Pulp-list at redhat.com <mailto:Pulp-list at redhat.com>
> https://www.redhat.com/mailman/listinfo/pulp-list
>
> ________________________________
>
> Information in this e-mail may be confidential. It is intended
> only for the addressee(s) identified above. If you are not the
> addressee(s), or an employee or agent of the addressee(s),
> please note that any dissemination, distribution, or copying of
> this communication is strictly prohibited. If you have received
> this e-mail in error, please notify the sender of the error.
>
> _______________________________________________
> Pulp-list mailing list
> Pulp-list at redhat.com <mailto:Pulp-list at redhat.com>
> https://www.redhat.com/mailman/listinfo/pulp-list
>
> _______________________________________________
> Pulp-list mailing list
> Pulp-list at redhat.com <mailto:Pulp-list at redhat.com>
> https://www.redhat.com/mailman/listinfo/pulp-list____
>
> __ __
>
>
> ------------------------------------------------------------------------
>
> Information in this e-mail may be confidential. It is intended only
> for the addressee(s) identified above. If you are not the
> addressee(s), or an employee or agent of the addressee(s), please
> note that any dissemination, distribution, or copying of this
> communication is strictly prohibited. If you have received this
> e-mail in error, please notify the sender of the error.
>
>
>
>
> _______________________________________________
> Pulp-list mailing list
> Pulp-list at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
>
More information about the Pulp-list
mailing list