[Pulp-list] Qpid SSL on Pulp 2.4

[Brian Bouterse]

Hi Jason,

I think the simplest description of the problem you have is that httpd won't start. It looks like httpd won't start because the settings in server.conf are not correct or point to files that don't provide the necessary aspects for SSL to work. The settings that qpidd uses (the server) are in qpidd.conf. Pulp keeps its settings that the webserver reads in /etc/pulp/server.conf. How did you configure server.conf? The pulp-qpid-ssl-cfg script also gives some recommended settings so refer to those also. It would be helpful if you provided a copy of your server.conf.

-Brian


----- Original Message -----
> From: "Jason Ashby (IMS)" <AshbyJ at imsweb.com>
> To: "Brian Bouterse" <bbouters at redhat.com>
> Cc: pulp-list at redhat.com
> Sent: Friday, October 24, 2014 11:11:50 AM
> Subject: RE: [Pulp-list] Qpid SSL on Pulp 2.4
> 
> Hi Brian,
> Thanks for the reply. For now, I'll ditch the troubleshooting with "openssl
> s_client" and focus on just the pulp commands.  Let me know if there are
> better commands to test with.
> 
> 
> $ pulp-admin tasks list
> +----------------------------------------------------------------------+
>                                  Tasks
> +----------------------------------------------------------------------+
> 
> The web server reported an error trying to access the Pulp application. The
> likely cause is that the pulp-manage-db script has not been run prior to
> starting the server. More information can be found in Apache's error log file
> on
> the server itself.
> 
> I've run this already, but for shitsngiggles I ran it again:
> 
> $ sudo -u apache pulp-manage-db
> Loading content types.
> /usr/lib/python2.6/site-packages/pulp/server/db/connection.py:133:
> DeprecationWarning: The safe parameter is deprecated. Please use write
> concern options instead.
>   return method(*args, **kwargs)
> Content types loaded.
> Ensuring the admin role and user are in place.
> Admin role and user are in place.
> Beginning database migrations.
> Migration package pulp.server.db.migrations is up to date at version 9
> Migration package pulp_puppet.plugins.migrations is up to date at version 2
> Migration package pulp_rpm.plugins.migrations is up to date at version 16
> Database migrations complete.
> $
> 
> "pulp-admin tasks list" results in same thing as above.  Here is some of
> /var/log/messages:
> 
> ...
> Oct 24 10:53:49 kiwi pulp: pulp.server.db.connection:INFO: Attempting
> Database connection with seeds = 127.0.0.1:27017
> Oct 24 10:53:49 kiwi pulp: pulp.server.db.connection:INFO: Connection
> Arguments: {'max_pool_size': 10}
> Oct 24 10:53:49 kiwi pulp: pulp.server.db.connection:INFO: Database
> connection established with: seeds = 127.0.0.1:27017, name = pulp_database
> Oct 24 10:53:49 kiwi pulp: pulp.server.async.scheduler:INFO: Worker Timeout
> Monitor Started
> Oct 24 10:53:49 kiwi qpidd[17032]: 2014-10-24 10:53:49 [System] error Error
> reading socket: Success(0)
> Oct 24 10:53:49 kiwi pulp: pulp.server.async.scheduler:ERROR: [Errno 1]
> _ssl.c:492: error:14090086:SSL
> routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
> Oct 24 10:53:49 kiwi qpidd[17032]: 2014-10-24 10:53:49 [System] error Error
> reading socket: Success(0)
> Oct 24 10:53:49 kiwi pulp: celery.beat:ERROR: beat: Connection error: [Errno
> 1] _ssl.c:492: error:14090086:SSL
> routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed. Trying again
> in 2.0 seconds...
> Oct 24 10:53:51 kiwi goferd: [WARNING][Thread-2] qpid.messaging:525 - trying:
> pulp.example.com:5671
> Oct 24 10:54:19 kiwi qpidd[17032]: 2014-10-24 10:54:19 [System] error Error
> reading socket: Success(0)
> ...
> 
> Apache logs below.  Wsgi seems to timeout when "pulp-admin tasks list" is
> run.
> 
> # /etc/httpd/logs/error_log
> ...
> [Fri Oct 24 11:02:20 2014] [error] Exception AttributeError: "'NoneType'
> object has no attribute 'close'" in <bound method PipeWaiter.__del__ of
> PipeWaiter(18, 19)> ignored
> [Fri Oct 24 11:02:20 2014] [error] Exception AttributeError: "'NoneType'
> object has no attribute 'close'" in <bound method PipeWaiter.__del__ of
> PipeWaiter(15, 17)> ignored
> [Fri Oct 24 11:02:21 2014] [notice] suEXEC mechanism enabled (wrapper:
> /usr/sbin/suexec)
> [Fri Oct 24 11:02:21 2014] [notice] Digest: generating secret for digest
> authentication ...
> [Fri Oct 24 11:02:21 2014] [notice] Digest: done
> [Fri Oct 24 11:02:21 2014] [notice] Apache/2.2.15 (Unix) DAV/2 mod_ssl/2.2.15
> OpenSSL/1.0.1e-fips mod_wsgi/3.4 Python/2.6.6 configured -- resuming normal
> operations
> 
> # /etc/httpd/logs/ssl_error_log.   x.x.x.x = my ip.  I'm testing from the
> actual pulp server's command line.
> [Fri Oct 24 11:07:05 2014] [error] [client x.x.x.x] Script timed out before
> returning headers: webservices.wsgi
> 
> Also I get these when I do a httpd restart, in case its helpful:
> 
> [Fri Oct 24 11:01:31 2014] [error] [client x.x.x.x] File does not exist:
> /var/www/pub/https/repos/centos/6.5/os
> [Fri Oct 24 11:01:31 2014] [error] [client x.x.x.x] Symbolic link not allowed
> or link target not accessible: /var/www/pub/https/repos/DuoSecurity
> [Fri Oct 24 11:01:31 2014] [error] [client x.x.x.x] Symbolic link not allowed
> or link target not accessible:
> /var/www/pub/https/repos/Prod-CentOS65-updates-i386 x.x.x.x.202.228.26]
> Symbolic link not allowed or link target not accessible:
> /var/www/pub/https/repos/Prod-CentOS65-updates-x86_64
> ...
> 
> Wsgi times out because of the failed cert verification?  "certificate verify
> failed" usually means the CA cert is not in the CA bundle on the system, but
> I've added my root and intermediary CA's to
> /etc/pki/tls/certs/ca-bundle.crt.  Does pulp/gofer/qpid refer to something
> else to verify certs?
> 
> 
> -----Original Message-----
> From: Brian Bouterse [mailto:bbouters at redhat.com]
> Sent: Friday, October 24, 2014 10:27 AM
> To: Ashby, Jason (IMS)
> Cc: pulp-list at redhat.com
> Subject: Re: [Pulp-list] Qpid SSL on Pulp 2.4
> 
> Hi Jason,
> 
> I've successfully used Qpid over SSL with Pulp 2.4 with port 5671. I see that
> you've configured the Qpid side of the SSL connection, but is server.conf
> for pulp configured? Refer to the docs [0] for how to configure the Pulp
> side of this.
> 
> The login portions of Pulp are unrelated to the tasking system. Using
> pulp-admin to login is not related to the tasking or consumer communications
> through Qpid.
> 
> Do the Pulp services start properly? I see you are testing with SSL, but I'm
> not seeing what Pulp commands you are running and what errors they produce.
> 
> [0]:
> http://pulp-user-guide.readthedocs.org/en/latest/broker-settings.html#qpid-with-ssl
> 
> -Brian
> 
> 
> 
> ----- Original Message -----
> > From: "Jason Ashby (IMS)" <AshbyJ at imsweb.com>
> > To: pulp-list at redhat.com
> > Sent: Friday, October 24, 2014 9:55:00 AM
> > Subject: [Pulp-list] Qpid SSL on Pulp 2.4
> >
> >
> >
> > Hi all,
> >
> >
> >
> > Apologies up front for the long email :). I just upgraded from Pulp 2.3 to
> > 2.4 and I’m having an issue with Qpid over SSL. Is anyone using Qpid over
> > SSL (port 5671) successfully in pulp 2.4? I don’t see much chatter about
> > it,
> > so I can’t find much info. I’m almost out of ideas for troubleshooting, so
> > any tips here are appreciated.
> >
> >
> >
> > I ran pulp-qpid-ssl-cfg and pointed to the same CA cert and key as I used
> > with Pulp 2.3. So I have the same configuration and certs as I did with 2.3
> > which worked fine (config listed further below). Anyway, I get the
> > following
> > ssl error when testing with openssl:
> >
> >
> >
> >
> >
> > $ openssl s_client -connect pulp.example.com:5671
> >
> >
> >
> > verify error:num=19:self signed certificate in certificate chain
> >
> > verify return:0
> >
> > 140594689586856:error:14094412:SSL routines:SSL3_READ_BYTES:sslv3 alert bad
> > certificate:s3_pkt.c:1292:SSL alert number 42
> >
> > 140594689586856:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake
> > failure:s23_lib.c:184:
> >
> >
> >
> > $ openssl s_client -connect pulp.example.com:5671 -tls1
> >
> >
> >
> > verify error:num=19:self signed certificate in certificate chain
> >
> > verify return:0
> >
> > 140594689586856:error:14094412:SSL routines:SSL3_READ_BYTES:sslv3 alert bad
> > certificate:s3_pkt.c:1292:SSL alert number 42
> >
> > 140594689586856:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake
> > failure:s23_lib.c:184:
> >
> >
> >
> > $ openssl s_client -connect pulp.example.com:5671 -tls1_1 # and same result
> > for -tls1_2
> >
> >
> >
> > CONNECTED(00000003)
> >
> > 139803025839784:error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version
> > number:s3_pkt.c:345:
> >
> >
> >
> >
> >
> > These are the same certificates used for the pulp server on apache 443,
> > which
> > is working fine. I’m using an intermediary certificate as the CA for pulp
> > and qpid. i.e. it’s a sub-CA that is signed by our company’s own root CA.
> > When pulp-qpid-ssl-cfg asks for the CA cert, I’ve tried both the sub-CA
> > cert
> > by itself and also a chain that includes the root + sub-CA certs. (The
> > chain
> > is what I’m currently using since the sub-CA cert by itself gives “unable
> > to
> > find local issuer certificate” because the root CA couldn’t be found.)
> >
> >
> >
> > ….My versions and configs…
> >
> >
> >
> > Pulp 2.4.3 server on CentOS 6.5
> >
> > Qpidd version 0.26
> >
> >
> >
> > # /etc/qpid/qpidd.conf
> >
> > auth=no
> >
> > require-encryption=yes
> >
> > ssl-require-client-authentication=yes
> >
> > ssl-cert-db=/etc/pki/pulp/qpid/nss
> >
> > ssl-cert-password-file=/etc/pki/pulp/qpid/nss/password
> >
> > ssl-cert-name=broker
> >
> > ssl-port=5671
> >
> > #
> >
> >
> >
> > When connecting with pulp-admin, I get a similar sslv3 certificate error
> > with
> > “certificate verify failed” for gofer.
> >
> >
> >
> > Perhaps this is this a supported protocol issue with Qpid or NSS? I can’t
> > see
> > how to specify supported protocols in the qpidd.conf file. I’m wondering if
> > NSS restricts protocols at all? I don’t know much about it.
> >
> >
> >
> > Thanks for any help,
> >
> > Jason
> >
> >
> >
> >
> >
> > Information in this e-mail may be confidential. It is intended only for the
> > addressee(s) identified above. If you are not the addressee(s), or an
> > employee or agent of the addressee(s), please note that any dissemination,
> > distribution, or copying of this communication is strictly prohibited. If
> > you have received this e-mail in error, please notify the sender of the
> > error.
> >
> > _______________________________________________
> > Pulp-list mailing list
> > Pulp-list at redhat.com
> > https://www.redhat.com/mailman/listinfo/pulp-list
> 
> ________________________________
> 
> Information in this e-mail may be confidential. It is intended only for the
> addressee(s) identified above. If you are not the addressee(s), or an
> employee or agent of the addressee(s), please note that any dissemination,
> distribution, or copying of this communication is strictly prohibited. If
> you have received this e-mail in error, please notify the sender of the
> error.
>