[katello-devel] Issues configuring latest build - Part 1

[Lukas Zapletal]

I've never seen this, but something is wrong with Puppet installer. The
latest (Foreman perhaps) changes changed the ordering and again - we
have various issues. Currently, this is third issue:

1. Seed fails because Tomcat is not yet running.
2. Pulp migrate fails because Mongo is not yet running.
3. Foreman database creation fails because (ditto).

Yesterday I pushed a fix:

https://github.com/Katello/katello/pull/631

It did not help with tomcat issue, but it seems to fix mongo issue. So I
prepared new pull request:

https://github.com/Katello/katello/pull/640

That adds time into the puppet log and also is adding additional 5
second sleep to both tomcat and postgres.

Several months ago I have introduced service-wait script that should
solve all those things. It is a wrapper around /sbin/service that
delegates all the stuff, but for the three above services it waits after
start and checks if the service is responding:

https://github.com/Katello/katello/blob/master/src/script/service-wait

For tomcat we was waiting until http://localhost:8080 was responding,
which was not working for all cases. Now I changed it to wait until
https://localhost:8443/candlepin/status responds - again today I found
it did not help.

For mongo we wait until mongo command evaluates a dummy expression.
Since mongo command connects to mongod using TCP, it should be enough.
Again, yesterday I had a report that pulp migration failed because -
believe it or not - mongo was not responding.

And finally for postgres we wait until port is opened with a netstat
command, because I was not able to come with a better solution.

Puppet was working, until we touched it and reordered things. Not it
strikes back.

At the moment, I have added some date/time information to the puppet log
to confirm all of this. I've also added 5 seconds sleep after start or
restart for all the three services just to check if this helps at all.

Please be patient, it's difficult to track those errors, because I am
not able to reproduce those race condition bugs everytime. It is also
possible I am tracking it wrong - maybe it's not about non-responding
services, maybe it is something completely different.

LZ

On Tue, Sep 11, 2012 at 06:25:12PM -0400, Og Maciel wrote:
> Before filing a BZ I wanted to check that this is not a known issue already:
> 
> 
> # katello-configure --deployment=katello
> Starting Katello configuration
> The top-level log file is [/var/log/katello/katello-configure-20120911-174543/main.log]
> err: /Stage[main]/Postgres::Service/Exec[wait-for-postgresql]: Failed to call refresh: Command exceeded timeout at /usr/share/katello/install/puppet/modules/postgres/manifests/service.pp:25
> Creating Candlepin database user
> 
>   Failed, please check [/var/log/katello/katello-configure/create-postgresql-candlepin-user.log]
>   Report errors using # katello-debug tool.
> Creating Katello database user
> 
>   Failed, please check [/var/log/katello/katello-configure/create-postgresql-katello-user.log]
>   Report errors using # katello-debug tool.
> err: /Stage[main]/Foreman::Config/Postgres::Createuser[foreman]/Sqlexec[createuser-foreman]/Exec[psql -h localhost --username=postgres postgres -c "CREATE ROLE foreman WITH LOGIN PASSWORD 'foreman' CREATEDB;" >> /var/log/katello/katello-configure/create-postgresql-foreman-user.log 2>&1]/returns: change from notrun to 0 failed: psql -h localhost --username=postgres postgres -c "CREATE ROLE foreman WITH LOGIN PASSWORD 'foreman' CREATEDB;" >> /var/log/katello/katello-configure/create-postgresql-foreman-user.log 2>&1 returned 2 instead of one of [0] at /usr/share/katello/install/puppet/modules/postgres/manifests/sqlexec.pp:16
> 
> Environment:
> 
> * candlepin-0.7.8.2-1.el6.noarch
> * candlepin-cert-consumer-qetello01.usersys.redhat.com-1.0-1.noarch
> * candlepin-selinux-0.7.8.2-1.el6.noarch
> * candlepin-tomcat6-0.7.8.2-1.el6.noarch
> * foreman-1.0.0-14.1648bec.el6.noarch
> * foreman-postgresql-1.0.0-14.1648bec.el6.noarch
> * katello-1.1.10-1.git.31.2fb829c.el6.noarch
> * katello-agent-1.1.2-1.el6.noarch
> * katello-all-1.1.10-1.git.31.2fb829c.el6.noarch
> * katello-candlepin-cert-key-pair-1.0-1.noarch
> * katello-certs-tools-1.1.8-1.el6.noarch
> * katello-cli-1.1.6-1.el6.noarch
> * katello-cli-common-1.1.6-1.el6.noarch
> * katello-common-1.1.10-1.git.31.2fb829c.el6.noarch
> * katello-configure-1.1.8-1.git.14.2d383ea.el6.noarch
> * katello-glue-candlepin-1.1.10-1.git.31.2fb829c.el6.noarch
> * katello-glue-foreman-1.1.10-1.git.31.2fb829c.el6.noarch
> * katello-glue-pulp-1.1.10-1.git.31.2fb829c.el6.noarch
> * katello-qpid-broker-key-pair-1.0-1.noarch
> * katello-qpid-client-key-pair-1.0-1.noarch
> * katello-selinux-1.1.1-1.el6.noarch
> * pulp-1.1.12-1.el6.noarch
> * pulp-common-1.1.12-1.el6.noarch
> * pulp-selinux-server-1.1.12-1.el6.noarch
> 
> 
> -- 
> Og Maciel
> 
> Senior QA Engineer
> Red Hat, Inc.
> +1.646.707.7723
> irc: omaciel
> 
> _______________________________________________
> katello-devel mailing list
> katello-devel at redhat.com
> https://www.redhat.com/mailman/listinfo/katello-devel

-- 
Later,

 Lukas "lzap" Zapletal
 #katello #systemengine