[Freeipa-users] ipa-getcert list fails to report correctly - RESOLVED
Les Stott
Less at imagine-sw.com
Wed Feb 25 02:11:48 UTC 2015
> -----Original Message-----
> From: freeipa-users-bounces at redhat.com [mailto:freeipa-users-
> bounces at redhat.com] On Behalf Of Les Stott
> Sent: Monday, 23 February 2015 8:01 PM
> To: Rob Crittenden; Martin Kosek; freeipa-users at redhat.com; Endi Dewata;
> Jan Cholasta
> Subject: Re: [Freeipa-users] ipa-getcert list fails to report correctly
>
>
>
> > -----Original Message-----
> > From: freeipa-users-bounces at redhat.com [mailto:freeipa-users-
> > bounces at redhat.com] On Behalf Of Les Stott
> > Sent: Monday, 23 February 2015 12:18 PM
> > To: Rob Crittenden; Martin Kosek; freeipa-users at redhat.com; Endi
> > Dewata; Jan Cholasta
> > Subject: Re: [Freeipa-users] ipa-getcert list fails to report
> > correctly
> >
> >
> >
> > > -----Original Message-----
> > > From: Rob Crittenden [mailto:rcritten at redhat.com]
> > > Sent: Saturday, 21 February 2015 1:39 AM
> > > To: Martin Kosek; Les Stott; freeipa-users at redhat.com; Endi Dewata;
> > > Jan Cholasta
> > > Subject: Re: [Freeipa-users] ipa-getcert list fails to report
> > > correctly
> > >
> > > Martin Kosek wrote:
> > > > On 02/20/2015 06:56 AM, Les Stott wrote:
> > > >> Hi all,
> > > >>
> > > >> The following is blocking the ability for me to install a CA replica.
> > > >>
> > > >> Environment:
> > > >>
> > > >> RHEL 6.6
> > > >>
> > > >> IPA 3.0.0-42
> > > >>
> > > >> PKI 9.0.3-38
> > > >>
> > > >> On the master the following is happening:
> > > >>
> > > >> ipa-getcert list
> > > >>
> > > >> Number of certificates and requests being tracked: 5.
> > > >>
> > > >> (but it shows no certificate details in the output)
> > > >>
> > > >> Running "getcert list" shows complete output.
> > > >>
> > > >> Also, when trying to browse
> > > >> https://master.mydomain.com/ca/ee/ca/getCertChain i get a failed
> > > >> response. The apache error logs on the master show....
> > > >>
> > > >> [Thu Feb 19 23:23:23 2015] [error] SSL Library Error: -12271 SSL
> > > >> client cannot verify your certificate
> > > >>
> > > >> The reason I am trying to browse that address is because that's
> > > >> what the ipa-ca-install setup is failing at (it complains that
> > > >> the CA certificate is not in proper format, in fact it's not able
> > > >> to get it at all).
> > > >>
> > > >> I know from another working ipa setup that ....
> > > >>
> > > >> Browsing to the above address provides valid xml content and
> > > >> ipa-getcert list shows certificate details and not just the
> > > >> number of tracked certificates.
> > > >>
> > > >> Been trying for a long time to figure out the issues without luck.
> > > >>
> > > >> I would greatly appreciate any help to troubleshoot and resolve
> > > >> the above issues.
> > > >>
> > > >> Regards,
> > > >>
> > > >> Les
> > > >
> > > > Endi or JanC, would you have any advise for Les? To me, it looks
> > > > like the Apache does not have proper certificate installed.
> > > >
> > > > My ipa-getcert on RHEL-6.6 shows 3 Server-Certs tracked, making it
> > > > in total of 8 certs tracked:
> > > >
> > > > # ipa-getcert list
> > > > Number of certificates and requests being tracked: 8.
> > > > Request ID '20141111000002':
> > > > status: MONITORING
> > > > stuck: no
> > > > key pair storage:
> > > > type=NSSDB,location='/etc/dirsrv/slapd-IDM-LAB-BOS-REDHAT-
> > > COM',nicknam
> > > > e='Server-Cert',token='NSS
> > > > Certificate
> > > > DB',pinfile='/etc/dirsrv/slapd-IDM-LAB-BOS-REDHAT-COM/pwdfile.txt'
> > > > certificate:
> > > > type=NSSDB,location='/etc/dirsrv/slapd-IDM-LAB-BOS-REDHAT-
> > > COM',nicknam
> > > > e='Server-Cert',token='NSS
> > > > Certificate DB'
> > > > CA: IPA
> > > > issuer: CN=Certificate Authority,O=EXAMPLE.COM
> > > > subject: CN=vm-086.example.com,O=EXAMPLE.COM
> > > > expires: 2016-11-11 00:00:01 UTC
> > > > key usage:
> > > > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
> > > > eku: id-kp-serverAuth,id-kp-clientAuth
> > > > pre-save command:
> > > > post-save command:
> > > > track: yes
> > > > auto-renew: yes
> > > > Request ID '20141111000047':
> > > > status: MONITORING
> > > > stuck: no
> > > > key pair storage:
> > > > type=NSSDB,location='/etc/dirsrv/slapd-PKI-IPA',nickname='Server-
> Cert'
> > > > ,token='NSS Certificate
> > > > DB',pinfile='/etc/dirsrv/slapd-PKI-IPA/pwdfile.txt'
> > > > certificate:
> > > > type=NSSDB,location='/etc/dirsrv/slapd-PKI-IPA',nickname='Server-
> Cert'
> > > > ,token='NSS
> > > > Certificate DB'
> > > > CA: IPA
> > > > issuer: CN=Certificate Authority,O=EXAMPLE.COM
> > > > subject: CN=vm-086.example.com,O=EXAMPLE.COM
> > > > expires: 2016-11-11 00:00:46 UTC
> > > > key usage:
> > > > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
> > > > eku: id-kp-serverAuth,id-kp-clientAuth
> > > > pre-save command:
> > > > post-save command:
> > > > track: yes
> > > > auto-renew: yes
> > > > Request ID '20141111000302':
> > > > status: MONITORING
> > > > stuck: no
> > > > key pair storage:
> > > > type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',toke
> > > > n= 'N SS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
> > > > certificate:
> > > > type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',toke
> > > > n=
> > > > 'N
> > > > SS
> > > > Certificate DB'
> > > > CA: IPA
> > > > issuer: CN=Certificate Authority,O=EXAMPLE.COM
> > > > subject: CN=vm-086.example.com,O=EXAMPLE.COM
> > > > expires: 2016-11-11 00:03:02 UTC
> > > > key usage:
> > > > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
> > > > eku: id-kp-serverAuth,id-kp-clientAuth
> > > > pre-save command:
> > > > post-save command:
> > > > track: yes
> > > > auto-renew: yes
> > > >
> > > >
> > > > What is actually in your Apache NSS database?
> > > >
> > > > # certutil -L -d /etc/httpd/alias/
> > > >
> > > > Martin
> > > >
> > >
> > > Remember ipa-getcert is just a shortcut for certificates using the
> > > certmonger CA named IPA, so it's more a filter than anything else. I
> > > don't know why it wouldn't display any output but I'd file a bug.
> > >
> > > I think we'd need to see the getcert list output to try to figure
> > > out what is going on.
> > >
> > > As for the SSL error fetching the cert chain I think Martin may be
> > > onto something. The request is proxied through Apache. I think the
> > > client here might be the Apache proxy client.
> > >
> > > I believe this command replicates what Apache is doing, you might
> > > give it a try on the master. This will get the chain directly from
> > > dogtag, bypassing
> > > Apache:
> > >
> > > $ curl -v --cacert /etc/ipa/ca.crt
> > > https://`hostname`:9444/ca/ee/ca/getCertChain
> > >
> > > rob
> >
> > Certutil shows....
> >
> > certutil -L -d /etc/httpd/alias/
> >
> > Certificate Nickname Trust Attributes
> >
> > SSL,S/MIME,JAR/XPI
> >
> > MYDOMAIN.COM IPA CA CT,C,C
> > ipaCert u,u,u
> > Signing-Cert u,u,u
> > Server-Cert u,u,u
> >
> > curl -v --cacert /etc/ipa/ca.crt
> > https://`hostname`:9444/ca/ee/ca/getCertChain
> > * About to connect() to `hostname` port 9444 (#0)
> > * Trying 192.168.1.1... connected
> > * Connected to `hostname` (192.168.1.1) port 9444 (#0)
> > * Initializing NSS with certpath: sql:/etc/pki/nssdb
> > * CAfile: /etc/ipa/ca.crt
> > CApath: none
> > * SSL connection using TLS_RSA_WITH_AES_128_CBC_SHA
> > * Server certificate:
> > * subject: CN=`hostname`,O=MYDOMAIN.COM
> > * start date: Dec 13 01:21:30 2013 GMT
> > * expire date: Dec 03 01:21:30 2015 GMT
> > * common name: `hostname`
> > * issuer: CN=Certificate Authority,O=MYDOMAIN.COM
> > > GET /ca/ee/ca/getCertChain HTTP/1.1
> > > User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7
> > > NSS/3.16.2.3 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> > > Host: `hostname`:9444
> > > Accept: */*
> > >
> > < HTTP/1.1 200 OK
> > < Server: Apache-Coyote/1.1
> > < Content-Type: application/xml
> > < Content-Length: 1434
> > < Date: Mon, 23 Feb 2015 01:04:29 GMT
> > <
> > <?xml version="1.0" encoding="UTF-8"
> >
> standalone="no"?><XMLResponse><Status>0</Status><ChainBase64>MIID
> >
> zwYJKoZIhvcNAQcCoIIDwDCCA7wCAQExADAPBgkqhkiG9w0BBwGgAgQAoII
> >
> DoDCCA5wwggKEoAMCAQICAQEwDQYJKoZIhvcNAQELBQAwOjEYMBYGA1U
> >
> EChMPREVSSVZBVElWRVMuQ09NMR4wHAYDVQQDExVDZXJ0aWZpY2F0ZSB
> >
> BdXRob3JpdHkwHhcNMTMxMjEzMDEyMTI5WhcNMzMxMjEzMDEyMTI5Wj
> >
> A6MRgwFgYDVQQKEw9ERVJJVkFUSVZFUy5DT00xHjAcBgNVBAMTFUNlcnRp
> >
> ZmljYXRlIEF1dGhvcml0eTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCg
> >
> gEBAMAA8EaYhmpjSA8o3/1kB/W1+0K6+FrwCS+njOgRtXhiTdmtSddXSDVxH
> >
> OafFwqN26BR+QRPZbbpJY70gP3SG8W+J6+c37PMVNshWz6UfChGt6ubgFxlS
> >
> TGUUre2Osr9I4C836MXpGJvRx2VDEuMUxv8j7B9iDRnTDglseqPqrMct2No4w
> >
> k4cLtA9puBJb0Es76SOHP9edXlf6GBnuYwR8YMc1yJLqpP8IGpHhEkVxMsRpqk
> >
> EpuuRwEFa7uBcTDhqVV24BpFlseZVubpiOdEgfb3IRBTjvI1Mum9OCJbuj9P/W
> >
> mqMnrA0sQsmF/R3WBwFdMAsN3+bQCRw73+rwoeDNcCAwEAAaOBrDCBq
> >
> TAfBgNVHSMEGDAWgBSO8J+j2jAuyg3a0yE+3oVCQJCWUTAPBgNVHRMBAf8
> >
> EBTADAQH/MA4GA1UdDwEB/wQEAwIBxjAdBgNVHQ4EFgQUjvCfo9owLsoN
> >
> 2tMhPt6FQkCQllEwRgYIKwYBBQUHAQEEOjA4MDYGCCsGAQUFBzABhipodHR
> > wOi8vc2I!
> > ybW9uMDEuZGVyaXZhdGl2ZXMuY29tOjgwL2* Connection #0 to host
> `hostname`
> > left intact
> > * Closing connection #0
> >
> NhL29jc3AwDQYJKoZIhvcNAQELBQADggEBAKH8YkoTAzX2xNYMkZSDK84EK3
> >
> e4FUixdXxc/EC5ehjrtaqXT1KT9Fl9DAF5/jYNKqgmEmtHnPGlfQ7/Y1ESdhEGcB
> >
> ZjU4qLe4HaFXuw5c9odDYxhtjQUd1g7ifY8SKOcHDCY+6Xx6F/rhFgzrXXMndn8
> >
> ZaYryctPoOAj/5INnLrJq8S4XyLmb2BHM4e1ORQbOhDi8xjhfK2veYXvIu55Brhp
> >
> RSS/goz5oSE8e+QE/H9afRmeV2+WkS/YDhSyoUDb7CYjklRuONzX3GopKtp1y
> >
> yLXQZnBFjCvIJvja0mo3ik3AXxSZuOwUIlV23U8CyPU/rDeiV00iUyA/fLvdkEtZkx
> > AA==</ChainBase64></XMLResponse>
> >
> >
> > In any event, I've decided to rebuilt my DR IPA environment. Late last
> > year the master in DR had to be rebuilt due to a disk issue. While IPA
> > was restored manually and appeared to be working fine, CA replication
> > hasn't worked. I finally got CA replication working in Prod after
> > enabling needed apache modules and performing a yum update to update
> > related packages, but these things didn't help in DR. It's my strong
> > suspicion that something got missed when restoring the DR master IPA
> > server and this is what is causing all my grief. Therefore, I'm going to wipe it
> out and start from scratch in DR.
> > There are other benefits for me to do this anyway.
> >
>
> Well things have gone from bad to worse.
>
> I removed IPA in DR. uninstalled all ipa clients, uninstalled replicas, removed
> replication agreements and removed the master. Ran pki-remove to clear
> any leftover pki instances and used certutil -D to remove left behind ipa
> entries in /etc/httpd/alias.
>
> So, clean slate to start again.
>
> This time, in order to mirror config with prod, I began an installation for the
> master on a different server, let's call it serverb. It was previously a replica (in
> my prod environment, serverb is the true master, servera, serverc, and
> serverd are replicas).
>
> So, trying to install a new fresh instance of IPA and it still fails to configure a
> CA.
>
> Attached is the relevant portion of the server install log file (ipa-server-
> install.txt). I have removed certificate and copyright info to reduce its size.
> Also my server to install is serverb.mydomain.com
>
> Apache logs at the time of the error show:
> [Mon Feb 23 03:05:31 2015] [error] SSL Library Error: -12195 Peer does not
> recognize and trust the CA that issued your certificate
>
> Certificate databases only show the following (note that "Server-Cert cert-
> pki-ca" got installed before the installer crashed). Prior to trying installation I
> had to manually remove server certs left behind from the previous
> installation via ...
> certutil -d /etc/httpd/alias -D -n "Server-Cert"
> certutil -d /etc/httpd/alias -D -n "MYDOMAIN.COM IPA CA"
> certutil -d /etc/httpd/alias -D -n ipaCert
>
> certutil -L -d /var/lib/pki-ca/alias
> Certificate Nickname Trust Attributes
> SSL,S/MIME,JAR/XPI
> Server-Cert cert-pki-ca CTu,Cu,Cu
>
> certutil -L -d /etc/pki/nssdb
> Certificate Nickname Trust Attributes
> SSL,S/MIME,JAR/XPI
>
>
> Selinux is in permissive mode.
> Ausearch -m avc does show some selinux issues, but its permissive mode so
> it should be ok right? In any event I have previously tried installing a CA
> replica with selinux disabled and it didn't help.
>
> I have tried removing ipa and pki rpms and reinstalling. Then rerunning the
> ipa server install script but the same error occurs.
>
> I noticed that /etc/ipa/ca.crt was still old, and referencing the original master.
> I removed that and again reran the installer but the same error occurred.
>
> Note also that /etc/ipa/cr.crt was not recreated when ipa-python was
> reinstalled.
>
> Other logs:
>
> /var/log/pki-ca/system shows
> 5042.main - [23/Feb/2015:03:05:12 EST] [3] [3] Cannot build CA chain. Error
> java.security.cert.CertificateException: Certificate is not a PKCS #11
> certificate 5042.main - [23/Feb/2015:03:05:12 EST] [13] [3] authz instance
> DirAclAuthz initialization failed and skipped, error=Property
> internaldb.ldapconn.port missing value
> 5042.http-9445-1 - [23/Feb/2015:03:05:26 EST] [3] [3] Cannot build CA chain.
> Error java.security.cert.CertificateException: Certificate is not a PKCS #11
> certificate
> 5042.http-9445-1 - [23/Feb/2015:03:05:35 EST] [3] [3] CASigningUnit: Object
> certificate not found. Error org.mozilla.jss.crypto.ObjectNotFoundException
>
> /var/log/pki-ca/catalina.out
> Feb 23, 2015 3:05:11 AM org.apache.catalina.startup.HostConfig
> deployDirectory
> INFO: Deploying web application directory ca 64-bit osutil library loaded 64-bit
> osutil library loaded CMS Warning: FAILURE: Cannot build CA chain. Error
> java.security.cert.CertificateException: Certificate is not a PKCS #11
> certificate|FAILURE: authz instance DirAclAuthz initialization failed and
> skipped, error=Property internaldb.ldapconn.port missing value| Server is
> started.
> Feb 23, 2015 3:05:12 AM org.apache.coyote.http11.Http11Protocol start
> INFO: Starting Coyote HTTP/1.1 on http-9180 Feb 23, 2015 3:05:12 AM
> org.apache.coyote.http11.Http11Protocol start
> INFO: Starting Coyote HTTP/1.1 on http-9443 Feb 23, 2015 3:05:12 AM
> org.apache.coyote.http11.Http11Protocol start
> INFO: Starting Coyote HTTP/1.1 on http-9445 Feb 23, 2015 3:05:12 AM
> org.apache.coyote.http11.Http11Protocol start
> INFO: Starting Coyote HTTP/1.1 on http-9444 Feb 23, 2015 3:05:12 AM
> org.apache.coyote.http11.Http11Protocol start
> INFO: Starting Coyote HTTP/1.1 on http-9446 Feb 23, 2015 3:05:12 AM
> org.apache.jk.common.ChannelSocket init
> INFO: JK: ajp13 listening on /0.0.0.0:9447 Feb 23, 2015 3:05:12 AM
> org.apache.jk.server.JkMain start
> INFO: Jk running ID=0 time=0/25 config=null Feb 23, 2015 3:05:12 AM
> org.apache.catalina.startup.Catalina start
> INFO: Server startup in 1655 ms
>
> I have no idea where to look next. There must be some remnant of the old
> system hanging around screwing things up but I cannot figure it out. This will
> drive me insane!
>
> I can provide more logs if needed.
>
> Thanks in advance for any help.
>
Have resolved this.
Here is the procedure to completely remove FreeIPA so you can start again.
ipa-server-install --uninstall
certutil -d /etc/httpd/alias -D -n "Server-Cert"
certutil -d /etc/httpd/alias -D -n "DERIVATIVES.COM IPA CA"
certutil -d /etc/httpd/alias -D -n ipaCert
certutil -d /etc/httpd/alias -D -n Signing-Cert
yum -y remove pki-selinux pki-ca pki-common pki-setup pki-silent pki-java-tools pki-symkey pki-util pki-native-tools ipa-server-selinux ipa-server ipa-client ipa-admintools ipa-python ipa-pki-ca-theme ipa-pki-common-theme 389-ds-base 389-ds-base-libs
userdel pkisrv
userdel pkiuser
rm -rf /etc/pki-ca /var/lib/pki-ca /var/log/pki-ca /etc/certmonger /etc/sysconfig/pki-ca /etc/sysconfig/pki /var/run/pki-ca.pid /usr/share/pki /etc/ipa /var/log/ipa*
reboot
Now you have a clean slate.
Then install works as normal for IPA Server, Replica and CA Replica installations.
Hope this saves someone else time in the future.
Regards,
Les
More information about the Freeipa-users
mailing list