[Freeipa-devel] [PATCH] 762 Let the framework be able to override the hostname

Thu Jun 23 09:12:38 UTC 2011

On Wed, 2011-06-22 at 08:51 -0400, Rob Crittenden wrote:
> Rob Crittenden wrote:
> > Martin Kosek wrote:
> >> On Fri, 2011-05-27 at 15:39 -0400, Rob Crittenden wrote:
> >>> Martin Kosek wrote:
> >>>> On Wed, 2011-05-25 at 11:29 -0400, Rob Crittenden wrote:
> >>>>> Martin Kosek wrote:
> >>>>>> On Fri, 2011-04-01 at 11:47 -0400, Rob Crittenden wrote:
> >>>>>>> The hostname is passed in during the server installation. We
> >>>>>>> should use
> >>>>>>> this hostname for the resulting server as well. It was being
> >>>>>>> discarded
> >>>>>>> and we always used the system hostname value.
> >>>>>>>
> >>>>>>> ticket 1052
> >>>>>>>
> >>>>>>> rob
> >>>>>>
> >>>>>> I have to NACK this again. I have a problem communicating with IPA
> >>>>>> on a
> >>>>>> master machine. I reproduced in on 2 different machines. Please,
> >>>>>> correct
> >>>>>> my steps if I am wrong, I do the following procedure
> >>>>>>
> >>>>>> 1) I prepare a fresh minimal F-15
> >>>>>> 2) Install freeipa-server (current master with your patches)
> >>>>>> 3) Add custom hostname to /etc/hosts
> >>>>>> 4) Install IPA server:
> >>>>>> ipa-server-install -p secret123 -a secret123 --hostname
> >>>>>> ipa.idm.lab.bos.redhat.com --setup-dns --forwarder=10.16.255.2
> >>>>>> 5) # kinit admin
> >>>>>> Password for admin at IDM.LAB.BOS.REDHAT.COM:
> >>>>>> 6) # ipa user-show admin
> >>>>>> ipa: ERROR: cannot connect to 'any of the configured servers':
> >>>>>> https://ipa.idm.lab.bos.redhat.com/ipa/xml,
> >>>>>> https://ipa.idm.lab.bos.redhat.com/ipa/xml
> >>>>>>
> >>>>>> # ping -c 1 ipa.idm.lab.bos.redhat.com
> >>>>>> PING ipa.idm.lab.bos.redhat.com (10.16.78.140) 56(84) bytes of data.
> >>>>>> 64 bytes from ipa.idm.lab.bos.redhat.com (10.16.78.140): icmp_req=1
> >>>>>> ttl=64 time=0.049 ms
> >>>>>>
> >>>>>> Apache error_log shows relevant errors:
> >>>>>>
> >>>>>> [Wed May 25 06:42:38 2011] [error] ipa: ERROR: Failed to start
> >>>>>> IPA: Unable to retrieve LDAP schema: Invalid credentials:
> >>>>>> SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.
> >>>>>> Minor code may provide more information (Permission denied)
> >>>>>> [Wed May 25 06:42:38 2011] [error] ipa: ERROR: Failed to start
> >>>>>> IPA: Unable to retrieve LDAP schema: Invalid credentials:
> >>>>>> SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.
> >>>>>> Minor code may provide more information (Permission denied)
> >>>>>> [Wed May 25 06:43:55 2011] [error] Exception KeyError:
> >>>>>> KeyError(140250828974112,) in<module 'threading' from
> >>>>>> '/usr/lib64/python2.7/threading.pyc'> ignored
> >>>>>> [Wed May 25 06:43:55 2011] [error] Exception KeyError:
> >>>>>> KeyError(140250828974112,) in<module 'threading' from
> >>>>>> '/usr/lib64/python2.7/threading.pyc'> ignored
> >>>>>> [Wed May 25 06:43:55 2011] [error] Exception KeyError:
> >>>>>> KeyError(140250828974112,) in<module 'threading' from
> >>>>>> '/usr/lib64/python2.7/threading.pyc'> ignored
> >>>>>> [Wed May 25 06:43:55 2011] [error] Exception KeyError:
> >>>>>> KeyError(140250828974112,) in<module 'threading' from
> >>>>>> '/usr/lib64/python2.7/threading.pyc'> ignored
> >>>>>> [Wed May 25 06:43:55 2011] [error] Exception KeyError:
> >>>>>> KeyError(140250828974112,) in<module 'threading' from
> >>>>>> '/usr/lib64/python2.7/threading.pyc'> ignored
> >>>>>> [Wed May 25 06:43:55 2011] [error] Exception KeyError:
> >>>>>> KeyError(140250828974112,) in<module 'threading' from
> >>>>>> '/usr/lib64/python2.7/threading.pyc'> ignored
> >>>>>> [Wed May 25 06:43:55 2011] [error] Exception KeyError:
> >>>>>> KeyError(140250828974112,) in<module 'threading' from
> >>>>>> '/usr/lib64/python2.7/threading.pyc'> ignored
> >>>>>> [Wed May 25 06:43:55 2011] [error] Exception KeyError:
> >>>>>> KeyError(140250828974112,) in<module 'threading' from
> >>>>>> '/usr/lib64/python2.7/threading.pyc'> ignored
> >>>>>> [Wed May 25 06:43:55 2011] [error] Exception KeyError:
> >>>>>> KeyError(140250828974112,) in<module 'threading' from
> >>>>>> '/usr/lib64/python2.7/threading.pyc'> ignored
> >>>>>> [Wed May 25 06:43:55 2011] [error] Exception KeyError:
> >>>>>> KeyError(140250828974112,) in<module 'threading' from
> >>>>>> '/usr/lib64/python2.7/threading.pyc'> ignored
> >>>>>> [Wed May 25 06:43:56 2011] [notice] caught SIGTERM, shutting down
> >>>>>> [Wed May 25 06:43:56 2011] [notice] SELinux policy enabled; httpd
> >>>>>> running as context system_u:system_r:kernel_t:s0
> >>>>>> [Wed May 25 06:43:57 2011] [notice] Digest: generating secret for
> >>>>>> digest authentication ...
> >>>>>> [Wed May 25 06:43:57 2011] [notice] Digest: done
> >>>>>> [Wed May 25 06:43:57 2011] [notice] Apache/2.2.17 (Unix) DAV/2
> >>>>>> mod_auth_kerb/5.4 mod_nss/2.2.17 NSS/3.12.9.0 mod_wsgi/3.2
> >>>>>> Python/2.7.1 configured -- resuming normal operations
> >>>>>> [Wed May 25 06:44:04 2011] [error] ipa: INFO: *** PROCESS START ***
> >>>>>> [Wed May 25 06:44:04 2011] [error] ipa: INFO: *** PROCESS START ***
> >>>>>> [Wed May 25 06:45:25 2011] [error] [client 10.16.78.140] mod_wsgi
> >>>>>> (pid=5192): Exception occurred processing WSGI script
> >>>>>> '/usr/share/ipa/wsgi.py'.
> >>>>>> [Wed May 25 06:45:25 2011] [error] [client 10.16.78.140] Traceback
> >>>>>> (most recent call last):
> >>>>>> [Wed May 25 06:45:25 2011] [error] [client 10.16.78.140] File
> >>>>>> "/usr/share/ipa/wsgi.py", line 48, in application
> >>>>>> [Wed May 25 06:45:25 2011] [error] [client 10.16.78.140] return
> >>>>>> api.Backend.session(environ, start_response)
> >>>>>> [Wed May 25 06:45:25 2011] [error] [client 10.16.78.140] File
> >>>>>> "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line
> >>>>>> 141, in __call__
> >>>>>> [Wed May 25 06:45:25 2011] [error] [client 10.16.78.140]
> >>>>>> self.create_context(ccache=environ.get('KRB5CCNAME'))
> >>>>>> [Wed May 25 06:45:25 2011] [error] [client 10.16.78.140] File
> >>>>>> "/usr/lib/python2.7/site-packages/ipalib/backend.py", line 110, in
> >>>>>> create_context
> >>>>>> [Wed May 25 06:45:25 2011] [error] [client 10.16.78.140]
> >>>>>> self.Backend.ldap2.connect(ccache=ccache)
> >>>>>> [Wed May 25 06:45:25 2011] [error] [client 10.16.78.140] File
> >>>>>> "/usr/lib/python2.7/site-packages/ipalib/backend.py", line 62, in
> >>>>>> connect
> >>>>>> [Wed May 25 06:45:25 2011] [error] [client 10.16.78.140] conn =
> >>>>>> self.create_connection(*args, **kw)
> >>>>>> [Wed May 25 06:45:25 2011] [error] [client 10.16.78.140] File
> >>>>>> "/usr/lib/python2.7/site-packages/ipalib/encoder.py", line 188, in
> >>>>>> new_f
> >>>>>> [Wed May 25 06:45:25 2011] [error] [client 10.16.78.140] return
> >>>>>> f(*new_args, **kwargs)
> >>>>>> [Wed May 25 06:45:25 2011] [error] [client 10.16.78.140] File
> >>>>>> "/usr/lib/python2.7/site-packages/ipaserver/plugins/ldap2.py",
> >>>>>> line 337, in create_connection
> >>>>>> [Wed May 25 06:45:25 2011] [error] [client 10.16.78.140]
> >>>>>> _handle_errors(e, **{})
> >>>>>> [Wed May 25 06:45:25 2011] [error] [client 10.16.78.140] File
> >>>>>> "/usr/lib/python2.7/site-packages/ipaserver/plugins/ldap2.py",
> >>>>>> line 118, in _handle_errors
> >>>>>> [Wed May 25 06:45:25 2011] [error] [client 10.16.78.140] raise
> >>>>>> errors.DatabaseError(desc=desc, info=info)
> >>>>>> [Wed May 25 06:45:25 2011] [error] [client 10.16.78.140]
> >>>>>> DatabaseError: Local error: SASL(-1): generic failure: GSSAPI
> >>>>>> Error: An invalid name was supplied (Hostname cannot be
> >>>>>> canonicalized)
> >>>>>> [Wed May 25 06:45:26 2011] [error] [client 10.16.78.140] mod_wsgi
> >>>>>> (pid=5193): Exception occurred processing WSGI script
> >>>>>> '/usr/share/ipa/wsgi.py'.
> >>>>>>
> >>>>>>
> >>>>>> You can check the problem on vm-140.idm.lab.bos.redhat.com if you
> >>>>>> want to.
> >>>>>>
> >>>>>> Martin
> >>>>>>
> >>>>>
> >>>>> The LDAP connection was still using the system hostname value. I
> >>>>> added a
> >>>>> conn.set_option(_ldap.OPT_HOST_NAME, api.env.host) in the two
> >>>>> places we
> >>>>> initialize an LDAP connection and that seems to have fixed it.
> >>>>>
> >>>>> Updated patch attached
> >>>>>
> >>>>> rob
> >>>>
> >>>> NACK. The problem on a master is gone. However, now ipa-replica-install
> >>>> is failing:
> >>>>
> >>>> # ipa-replica-install
> >>>> /home/mkosek/replica-info-vm-027.idm.lab.bos.redhat.com.gpg
> >>>> Directory Manager (existing master) password:
> >>>>
> >>>> creation of replica failed: Can't contact LDAP server:
> >>>>
> >>>>
> >>>> I found out that the root cause of the failure is in the change you
> >>>> just
> >>>> made in ldap2.py:
> >>>>
> >>>> def create_connection(self, ccache=None, bind_dn='', bind_pw='',
> >>>> tls_cacertfile=None, tls_certfile=None, tls_keyfile=None,
> >>>> debug_level=0):
> >>>> ...
> >>>> try:
> >>>> conn = _ldap.initialize(self.ldap_uri)
> >>>> conn.set_option(_ldap.OPT_HOST_NAME, api.env.host)<--
> >>>> if ccache is not None:
> >>>> os.environ['KRB5CCNAME'] = ccache
> >>>> ...
> >>>>
> >>>> because api.env.host points to the local host and not the remote
> >>>> master.
> >>>> When I commented this line out, installation continued OK. Then, it
> >>>> crashed again with our "favorite" dogtag's "invalid clone_uri"
> >>>> exception.
> >>>>
> >>>> Since we see this error also in other scenarios (not only custom
> >>>> --hostname) and the root cause is not in your patch I can ACK you patch
> >>>> 762 once the replica install bug is fixed.
> >>>>
> >>>> Martin
> >>>>
> >>>
> >>> Fixed both of these. We only need to set the hostname when using an
> >>> ldapi URI, so fixed both of those.
> >>>
> >>> I also fixed the Invalid clone_uri bug. The problem was we weren't
> >>> passing our new hostname to pkicreate so it was creating a CA for
> >>> whatever the value of `hostname` was. There is an environment variable
> >>> in pkicreate to pass in the hostname and doing that has fixed the
> >>> problem.
> >>>
> >>> rob
> >>
> >> Yes, this issue was fixed. It's good you find a way how to deal with
> >> clone_uri problem. However, I still hit some issues:
> >>
> >> 1) I think we have some Kerberos related problems when the custom
> >> hostname is used (ipa.idm.lab.bos.redhat.com on a
> >> vm-096.idm.lab.bos.redhat.com). Named and SSSD refuses to start on the
> >> system.
> >>
> >> /var/log/messages:
> >> May 30 05:04:35 vm-096 named[13932]: listening on IPv4 interface eth0,
> >> 10.16.78.96#53
> >> May 30 05:04:35 vm-096 named[13932]: generating session key for
> >> dynamic DNS
> >> May 30 05:04:36 vm-096 named[13932]: Failed to init credentials
> >> (Preauthentication failed)
> >> May 30 05:04:36 vm-096 named[13932]: loading configuration: failure
> >> May 30 05:04:36 vm-096 named[13932]: exiting (due to fatal error)
> >> May 30 05:04:36 vm-096 systemd[1]: named.service: control process
> >> exited, code=exited status=7
> >> May 30 05:04:36 vm-096 systemd[1]: Unit named.service entered failed
> >> state.
> >> May 30 05:07:41 vm-096 sssd: Starting up
> >> May 30 05:07:41 vm-096 sssd[be[idm.lab.bos.redhat.com]]: Starting up
> >> May 30 05:07:41 vm-096 sssd[be[idm.lab.bos.redhat.com]]: Error
> >> processing keytab file [(null)]: Principal
> >> [host/vm-096.idm.lab.bos.redhat.com at IDM.LAB.BOS.REDHAT.COM] was not
> >> found. Unable to create GSSAPI-encrypted LDAP connection.
> >
> > For the named issue I filed a bug against bind-dyndb-ldap for this,
> > https://bugzilla.redhat.com/show_bug.cgi?id=710261
> >
> > This is a similar problem I ran into where when you do an ldapi bind it
> > defaults to using the system hostname value.
> >
> > To fix the sssd problem we just need to set the ipa_hostname option
> > (they have lots of nice tuning options!). We just need to decide if we
> > always set this value or only at install time when the hostnames differ.
> >
> >> 2) My dogtag powered replica still refuses to install (happened to me on
> >> 2 fresh VMs) with "creation of replica failed: Configuration of CA
> >> failed".
> >>
> >> I investigated the ipareplica-install.log, I found a error that may be
> >> relevant. Maybe Ade will recognize some of them.
> >>
> >> #############################################
> >> Attempting to connect to: vm-028.idm.lab.bos.redhat.com:9445
> >> Connected.
> >> Posting Query =
> >> https://vm-028.idm.lab.bos.redhat.com:9445//ca/admin/console/config/wizard?p=9&op=next&xml=true&host=vm-028.idm.lab.bos.redhat.com&port=7389&binddn=cn%3DDirectory+Manager&__bindpwd=XXXXXXXX&basedn=o%3Dipaca&database=ipaca&display=%24displayStr&cloneStartTLS=on
> >>
> >> RESPONSE STATUS: HTTP/1.1 200 OK
> >> RESPONSE HEADER: Server: Apache-Coyote/1.1
> >> RESPONSE HEADER: Content-Type: application/xml;charset=UTF-8
> >> RESPONSE HEADER: Date: Mon, 30 May 2011 11:26:29 GMT
> >> RESPONSE HEADER: Connection: close
> >> ...
> >> <response>
> >> <panel>admin/console/config/databasepanel.vm</panel>
> >> <clone>clone</clone>
> >> <res/>
> >> <portStr>7389</portStr>
> >> <bindpwd>(sensitive)</bindpwd>
> >> <cloneStartTLS>on</cloneStartTLS>
> >> <hostname>vm-028.idm.lab.bos.redhat.com</hostname>
> >> <errorString>Master and clone should have the same base DN</errorString>
> >>
> >>
> >> The CA installation fails few error messages later.
> >>
> >> Providing excerpt of CA logs as they may be relevant:
> >>
> >> /var/log/pki-ca/catalina.out:
> >> ...
> >> CMS Warning: FAILURE: Cannot build CA chain. Error
> >> java.security.cert.CertificateException: Certificate is not a PKCS #11
> >> certificate|FAILURE: authz instance DirAclAuthz initialization failed
> >> and skipped, error=Property internaldb.ldapconn.port missing value|
> >> ...
> >> [Fatal Error] :2:15: Open quote is expected for attribute "BGCOLOR"
> >> associated with an element type "BODY".
> >>
> >> /var/log/pki-ca/system:
> >> 2893.main - [30/May/2011:07:25:47 EDT] [3] [3] Cannot build CA chain.
> >> Error java.security.cert.CertificateException: Certificate is not a
> >> PKCS #11 certificate
> >> 2893.main - [30/May/2011:07:25:47 EDT] [13] [3] authz instance
> >> DirAclAuthz initialization failed and skipped, error=Property
> >> internaldb.ldapconn.port missing value
> >>
> >> Martin
> >>
> >
> > Haven't had a chance to explore this one yet. It sure would be nice if
> > dogtag would tell us what the two differing base DNs are though...
> 
> This patch should resolve the remaining issues. It requires a patch to 
> bind-dyndb-ldap, I have a candidate patch in 
> https://bugzilla.redhat.com/show_bug.cgi?id=710261
> 
> rob

Hmm, good work there. Bind, SSSD on custom-hostname IPA master is
working now. IPA client and CA-powered replica too.

I found only one issue - ipactl is not working because it uses
socket.gethostname() instead of api.env.host. So if you fix this
one-liner its ACK from me.

Martin