[Pulp-list] Verifying component services when creating a new Pulp service

[Mark Lamourine]

----- Original Message -----
> Hi Mark,
> 
> The resource_manager and all pulp_workers emit celery heartbeats which are
> monitored by celerybeat component. You could subscribe to these heartbeats
> also through the Qpid broker and process them, but you could get a lot of
> assurance through easier means.
> 
> Each pulp worker has a name that starts with 'reserved_resource_worker-' and
> then a number. The resource manager has a name that starts with
> 'resource_manager'. Both of these components automatically create a queue on
> the broker that starts with their name and ends in '.dq'. Once the process
> disconnects from the broker for any reason the queue is automatically
> deleted by the broker. Using this reasoning you list the queues that are
> available on the broker using `qpid-stat -q` and you should see them. I've
> included my output [0] as an example.
> 
> The pulp_celerybeat component has a similar behavior, but it creates and
> deletes with a queue that starts with 'celeryev' and ends in a uuid. This is
> the queue that is monitoring the celery events, one of which is a heartbeat.
> 
> For verifying the httpd servers you could use wget to verify they are online
> and functioning, although I don't have a specific recommendation for that.
> 
> The database is less useful for determining which components are actually
> running because those components "work out of" the database, but don't
> necessarily note their own existence in the database. You should at least
> have a pulp_database inside MongoDB created.

Thanks Brian, this is exactly what I was looking for.  I actually got as far in my scripting as finding the celeryev* queue, but I wasn't sure what it meant. With this I can write up something that gets me a status quickly for each component.

I hadn't noted the apache service because that is pretty easy and it's the last thing I'll put up.  Is there one or more stock URLs that would be good "hello world" tests for the content and admin services? I've used the login URL for admin, but if there's something better....

Thanks again,

- Mark

> 
> -Brian
> 
> [0]:  http://ur1.ca/hz95m
> 
> 
> ----- Original Message -----
> > From: "Mark Lamourine" <markllama at redhat.com>
> > To: pulp-list at redhat.com
> > Sent: Thursday, August 14, 2014 10:31:27 AM
> > Subject: [Pulp-list] Verifying component services when creating a new Pulp
> > 	service
> > 
> > I'm working to get a Pulp service built in Docker containers and created
> > using Kubernetes.  I'm building on Aaron Weitekamp's initial docker work.
> > 
> > When putting a Pulp service up in containers it is important to know the
> > requirements and boundaries of each component service.  I'd like to have a
> > way to verify each service component before moving on to the next.
> > 
> > With MongoDB and QPID/AMQP, I know how do do this:
> > 
> > Mongo:
> > 
> >   mongo <hostname or IP address>
> >   > show dbs
> > 
> > I can verify a connnection to the mongodb server, confirm that the pulp
> > database has or has not been created.
> > 
> > QPID:
> > 
> > For a qpid service I can connect and send/recieve a message.
> > 
> > https://gist.github.com/markllama/e0a5db776d2a1c876ff6
> > 
> >   test_qpid.py <hostname or IP address>
> > 
> > 
> > What I'd like is some incantation to check that the pulp components have
> > connected to the database.  Do they heartbeat? The same for QPID.   Do they
> > create a set
> > of topics or queues who's presence and activity would indicate that the
> > components have connected?  Do the pulp services have the equivalent of an
> > echo or ping message which could be used to query the connected processes?
> > 
> > 
> > 1) Mongo service exists
> >  a) can connect
> >  b) database initialized
> >  c) * celerybeat service connected, heartbeating ??
> > 
> > 2) QPID service exists
> >  a) can connect
> >  b) can echo
> > 
> > 3) Celery beat
> >  a) Mongodb: db exists and is initialized
> >  a) Mongodb: Beat heartbeats?
> >  b) QPID:  Pulp topics/queues exist
> >  c) QPID:  celery beat service connected, heartbeating ??
> >  d) QPID:  celery beat service responds to echo request
> > 
> > 4) Resource Manager worker
> >  ...
> > 
> > 5) Apache server - content
> >  ...
> > 
> > 6) Apache server - admin
> >  ...
> > 
> > 
> > What I'm really looking for is an internal behavior profile for a working
> > service.
> > 
> > I'd also like a comprehensive list of the expected startup and lifecycle
> > syslog entries for each service.  When a service starts and attaches to
> > mongo/qpid/beat what would the signatures of success look like?  What would
> > be signatures of common failure modes?
> > 
> > Does anything like this exist?  Pointers?
> > 
> > How are people verifying the operation of the service both new and over its
> > life-cycle?
> > 
> > - Mark
> > 
> > --
> > Mark Lamourine <mlamouri at redhat.com>
> > Sr. Software Developer, Cloud Strategy
> > Red Hat, 314 Littleton Road, Westford MA 01886
> > Voice: +1 978 392 1093
> > http://people.redhat.com/~mlamouri
> > markllama @ irc://irc.freenod.org*lopsa
> > 
> > _______________________________________________
> > Pulp-list mailing list
> > Pulp-list at redhat.com
> > https://www.redhat.com/mailman/listinfo/pulp-list
> > 
> 
> _______________________________________________
> Pulp-list mailing list
> Pulp-list at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
> 

-- 
Mark Lamourine <mlamouri at redhat.com>
Sr. Software Developer, Cloud Strategy
Red Hat, 314 Littleton Road, Westford MA 01886
Voice: +1 978 392 1093
http://people.redhat.com/~mlamouri
markllama @ irc://irc.freenod.org*lopsa