[Freeipa-users] errors when one ipa server down
Dmitri Pal
dpal at redhat.com
Mon Sep 17 15:17:47 UTC 2012
On 09/17/2012 10:27 AM, Michael Mercier wrote:
> On 2012-09-10, at 4:35 AM, Petr Spacek wrote:
>
>> On 09/08/2012 05:03 PM, Dmitri Pal wrote:
>>> On 09/07/2012 04:50 PM, Rob Crittenden wrote:
>>>> Michael Mercier wrote:
>>>>> On 2012-09-07, at 2:47 PM, Dmitri Pal wrote:
>>>>>
>>>>>> On 09/07/2012 12:42 PM, Michael Mercier wrote:
>>>>>>> On 2012-09-07, at 12:14 PM, Dmitri Pal wrote:
>>>>>>>
>>>>>>>> On 09/06/2012 10:40 AM, Michael Mercier wrote:
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> I have experienced some odd connectivity issues using MMR with
>>>>>>>>> FreeIPA (all systems CentOS 6.3). I have 2 ipa servers
>>>>>>>>> (ipaserver / ipaserver2) setup using MMR.
>>>>>>>>>
>>>>>>>>> [root at ipaserver ~]#ipa-replica-manage list
>>>>>>>>> ipaserver.mpls.local: master
>>>>>>>>> ipaserver2.mpls.local: master
>>>>>>>>> [root at ipaserver ~]# rpm -qa|grep ipa
>>>>>>>>> libipa_hbac-1.8.0-32.el6.x86_64
>>>>>>>>> ipa-admintools-2.2.0-16.el6.x86_64
>>>>>>>>> ipa-server-2.2.0-16.el6.x86_64
>>>>>>>>> ipa-pki-ca-theme-9.0.3-7.el6.noarch
>>>>>>>>> libipa_hbac-python-1.8.0-32.el6.x86_64
>>>>>>>>> ipa-client-2.2.0-16.el6.x86_64
>>>>>>>>> ipa-server-selinux-2.2.0-16.el6.x86_64
>>>>>>>>> ipa-pki-common-theme-9.0.3-7.el6.noarch
>>>>>>>>> python-iniparse-0.3.1-2.1.el6.noarch
>>>>>>>>> ipa-python-2.2.0-16.el6.x86_64
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> [root at ipaserver2 ~]#ipa-replica-manage list
>>>>>>>>> ipaserver.mpls.local: master
>>>>>>>>> ipaserver2.mpls.local: master
>>>>>>>>> [root at ipaserver2 ~]# rpm -qa|grep ipa
>>>>>>>>> ipa-client-2.2.0-16.el6.x86_64
>>>>>>>>> ipa-server-2.2.0-16.el6.x86_64
>>>>>>>>> ipa-pki-ca-theme-9.0.3-7.el6.noarch
>>>>>>>>> ipa-python-2.2.0-16.el6.x86_64
>>>>>>>>> libipa_hbac-1.8.0-32.el6.x86_64
>>>>>>>>> python-iniparse-0.3.1-2.1.el6.noarch
>>>>>>>>> libipa_hbac-python-1.8.0-32.el6.x86_64
>>>>>>>>> ipa-admintools-2.2.0-16.el6.x86_64
>>>>>>>>> ipa-server-selinux-2.2.0-16.el6.x86_64
>>>>>>>>> ipa-pki-common-theme-9.0.3-7.el6.noarch
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> [mike at ipaclient ~]$ rpm -qa|grep ipa
>>>>>>>>> ipa-admintools-2.2.0-16.el6.x86_64
>>>>>>>>> python-iniparse-0.3.1-2.1.el6.noarch
>>>>>>>>> ipa-python-2.2.0-16.el6.x86_64
>>>>>>>>> libipa_hbac-python-1.8.0-32.el6.x86_64
>>>>>>>>> ipa-client-2.2.0-16.el6.x86_64
>>>>>>>>> libipa_hbac-1.8.0-32.el6.x86_64
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I have a webserver (zenoss) using kerberos authentication.
>>>>>>>>>
>>>>>>>>> [root at zenoss ~]# rpm -qa|grep ipa
>>>>>>>>> libipa_hbac-1.8.0-32.el6.x86_64
>>>>>>>>> libipa_hbac-python-1.8.0-32.el6.x86_64
>>>>>>>>> ipa-python-2.2.0-16.el6.x86_64
>>>>>>>>> ipa-client-2.2.0-16.el6.x86_64
>>>>>>>>> python-iniparse-0.3.1-2.1.el6.noarch
>>>>>>>>> ipa-admintools-2.2.0-16.el6.x86_64
>>>>>>>>>
>>>>>>>>> <Location />
>>>>>>>>> SSLRequireSSL
>>>>>>>>> AuthType Kerberos
>>>>>>>>> AuthName "Kerberos Login"
>>>>>>>>>
>>>>>>>>> KrbMethodK5Passwd Off
>>>>>>>>> KrbAuthRealms MPLS.LOCAL
>>>>>>>>> KrbSaveCredentials on
>>>>>>>>> KrbServiceName HTTP
>>>>>>>>> Krb5KeyTab /etc/http/conf.d/http.keytab
>>>>>>>>>
>>>>>>>>> AuthLDAPUrl "ldap://ipaserver.mpls.local
>>>>>>>>> ipaserver2.mpls.local/dc=mpls,dc=local?krbPrincipalName"
>>>>>>>>> RequestHeader set X_REMOTE_USER %{remoteUser}e
>>>>>>>>> require ldap-group
>>>>>>>>> cn=zenuser,cn=groups,cn=accounts,dc=mpls,dc=local
>>>>>>>>> </Location>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> With both ipaserver and ipaserver2 'up', if I connect to
>>>>>>>>> https://zenoss.mpls.local from ipaclient using firefox, I am
>>>>>>>>> successfully connected. If on ipaserver I do a 'ifdown eth0' and
>>>>>>>>> attempt another connection, it fails. I have also noticed the
>>>>>>>>> following:
>>>>>>>>>
>>>>>>>>> 1. I am unable to use the ipaserver2 management interface when
>>>>>>>>> ipaserver is unavailable.
>>>>>>>>> 2. It takes a longer period of time to do a kinit
>>>>>>>>>
>>>>>>>>> If the I then perform:
>>>>>>>>> [root at ipaserver ~]#ifup eth0
>>>>>>>>>
>>>>>>>>> [root at ipaserver2 ~]#ifdown eth0
>>>>>>>>>
>>>>>>>>> [mike at ipaclient ~]$kinit
>>>>>>>>> kinit: Cannot contact any KDC for realm 'MPLS.LOCAL' while
>>>>>>>>> getting initial credentials
>>>>>>>>>
>>>>>>>>> [root at ipaserver2 ~]#ifup eth0
>>>>>>>>>
>>>>>>>>> [mike at ipaclient ~]$ kinit
>>>>>>>>> Password for mike at MPLS.LOCAL:
>>>>>>>>> [mike at ipaclient ~]$
>>>>>>>>>
>>>>>>>>> [root at ipaserver2 ~]#ifdown eth0
>>>>>>>>>
>>>>>>>>> .. wait number of minutes
>>>>>>>>>
>>>>>>>>> ipaclient screen locks - type password - after a short delay (~7
>>>>>>>>> seconds) screen unlock compeletes
>>>>>>>>>
>>>>>>>>> [mike at ipaclient ~]$kinit
>>>>>>>>> Password for mike at MPLS.LOCAL:
>>>>>>>>> [mike at ipaclient ~]$
>>>>>>>>>
>>>>>>>>> Any ideas?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Mike
>>>>>>>> This seems to be some DNS problem.
>>>>>>>> You client does not see the second replica and might have some name
>>>>>>>> resolution timeouts.
>>>>>>>>
>>>>>>>> Please check your dns setup and krb5.conf on the client.
>>>>>>>>
>>>>>>>> To help more we need more details about you client configuration
>>>>>>>> DNS and
>>>>>>>> kerberos.
>>>>>>> Hi,
>>>>>>>
>>>>>>> Additional information...
>>>>>>>
>>>>>>> [root at zenoss ~]#more /etc/resolv.conf
>>>>>>> search mpls.local
>>>>>>> domain mpls.local
>>>>>>> nameserver 172.16.112.5
>>>>>>> nameserver 172.16.112.8
>>>>>>>
>>>>>>> [root at zenoss ~]# more /etc/krb5.conf
>>>>>>> #File modified by ipa-client-install
>>>>>>>
>>>>>>> [libdefaults]
>>>>>>> default_realm = MPLS.LOCAL
>>>>>>> dns_lookup_realm = true
>>>>>>> dns_lookup_kdc = true
>>>>>>> rdns = false
>>>>>>> ticket_lifetime = 24h
>>>>>>> forwardable = yes
>>>>>>>
>>>>>>> [realms]
>>>>>>> MPLS.LOCAL = {
>>>>>>> pkinit_anchors = FILE:/etc/ipa/ca.crt
>>>>>>> }
>>>>>>>
>>>>>>> [domain_realm]
>>>>>>> .mpls.local = MPLS.LOCAL
>>>>>>> mpls.local = MPLS.LOCAL
>>>>>>>
>>>>>>> [root at ipaclient ~]# more /etc/resolv.conf
>>>>>>> # Generated by NetworkManager
>>>>>>> search mpls.local
>>>>>>> nameserver 172.16.112.5
>>>>>>> nameserver 172.16.112.8
>>>>>>>
>>>>>>> [root at ipaclient ~]# more /etc/krb5.conf
>>>>>>> #File modified by ipa-client-install
>>>>>>>
>>>>>>> [libdefaults]
>>>>>>> default_realm = MPLS.LOCAL
>>>>>>> dns_lookup_realm = true
>>>>>>> dns_lookup_kdc = true
>>>>>>> rdns = false
>>>>>>> ticket_lifetime = 24h
>>>>>>> forwardable = yes
>>>>>>>
>>>>>>> [realms]
>>>>>>> MPLS.LOCAL = {
>>>>>>> pkinit_anchors = FILE:/etc/ipa/ca.crt
>>>>>>> }
>>>>>>>
>>>>>>> [domain_realm]
>>>>>>> .mpls.local = MPLS.LOCAL
>>>>>>> mpls.local = MPLS.LOCAL
>>>>>>>
>>>>>>> [root at ipaclient ~]# nslookup ipaserver
>>>>>>> Server: 172.16.112.5
>>>>>>> Address: 172.16.112.5#53
>>>>>>>
>>>>>>> Name: ipaserver.mpls.local
>>>>>>> Address: 172.16.112.5
>>>>>>>
>>>>>>> [root at ipaserver ~]#ifdown eth0
>>>>>>>
>>>>>>> [root at ipaclient ~]# nslookup ipaserver
>>>>>>> Server: 172.16.112.8
>>>>>>> Address: 172.16.112.8#53
>>>>>>>
>>>>>>> Name: ipaserver.mpls.local
>>>>>>> Address: 172.16.112.5
>>>>>>>
>>>>>>> [root at ipaclient ~]# nslookup ipaserver2
>>>>>>> Server: 172.16.112.8
>>>>>>> Address: 172.16.112.8#53
>>>>>>>
>>>>>>> Name: ipaserver2.mpls.local
>>>>>>> Address: 172.16.112.8
>>>>>>>
>>>>>>> Copy/paste from the DNS page on ipaserver/ipaserver2
>>>>>>>
>>>>>>> @ NS ipaserver.mpls.local.
>>>>>>> NS ipaserver2.mpls.local.
>>>>>>> _kerberos TXT MPLS.LOCAL
>>>>>>> _kerberos-master._tcp SRV 0 100 88 ipaserver
>>>>>>> SRV 0 100 88 ipaserver2
>>>>>>> _kerberos-master._udp SRV 0 100 88 ipaserver
>>>>>>> SRV 0 100 88 ipaserver2
>>>>>>> _kerberos._tcp SRV 0 100 88 ipaserver
>>>>>>> SRV 0 100 88 ipaserver2
>>>>>>> _kerberos._udp SRV 0 100 88 ipaserver
>>>>>>> SRV 0 100 88 ipaserver2
>>>>>>> _kpasswd._tcp SRV 0 100 464 ipaserver
>>>>>>> SRV 0 100 464 ipaserver2
>>>>>>> _kpasswd._udp SRV 0 100 464 ipaserver
>>>>>>> SRV 0 100 464 ipaserver2
>>>>>>> _ldap._tcp SRV 0 100 389 ipaserver
>>>>>>> SRV 0 100 389 ipaserver2
>>>>>>> _ntp._udp SRV 0 100 123 ipaserver
>>>>>>> SRV 0 100 123 ipaserver2
>>>>>>> ipaclient A 172.16.112.9
>>>>>>> ipaclient2 A 172.16.112.145
>>>>>>> ipaserver A 172.16.112.5
>>>>>>> ipaserver2 A 172.16.112.8
>>>>>>> zenoss A 172.16.112.6
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Mike
>>>>>>>
>>>>>> I noticed that there is no domain line in the resolv.conf on the
>>>>>> client.
>>>>>> AFAIU in this case it would determine the domain by the gethostname and
>>>>>> in case of network being down it will fail over to the hosts file.
>>>>>> I wonder what is in your /etc/hosts?
>>>>>> Dose it have just a short host name?
>>>>> [root at ipaclient ~]# more /etc/hosts
>>>>> 127.0.0.1 localhost.localdomain localhost
>>>>> ::1 localhost6.localdomain6 localhost6
>>>>>
>>>>>
>>>>> Add domain mpls.local to /etc/resolv.conf
>>>>>
>>>>> [root at ipaserver ~]#ifdown eth0
>>>>>
>>>>> [root at ipaclient ~]# kinit mike
>>>>> kinit: Cannot contact any KDC for realm 'MPLS.LOCAL' while getting
>>>>> initial credentials
>>>>> [root at ipaclient ~]# nslookup ipaserver
>>>>> Server: 172.16.112.8
>>>>> Address: 172.16.112.8#53
>>>>>
>>>>> Name: ipaserver.mpls.local
>>>>> Address: 172.16.112.5
>>>>>
>>>>> [root at ipaclient ~]# nslookup ipaserver2
>>>>> Server: 172.16.112.8
>>>>> Address: 172.16.112.8#53
>>>>>
>>>>> Name: ipaserver2.mpls.local
>>>>> Address: 172.16.112.8
>>>>>
>>>>> add '172.16.112.9 ipaclient.mpls.local ipaclient' to /etc/hosts
>>>>>
>>>>> [root at ipaserver ~]#ifup eth0
>>>>>
>>>>> [root at ipaclient ~]# kinit mike
>>>>> Password for mike at MPLS.LOCAL:
>>>>>
>>>>> [root at ipaserver ~]#ifdown eth0
>>>>>
>>>>> [root at ipaclient ~]# kinit mike
>>>>> kinit: Cannot contact any KDC for realm 'MPLS.LOCAL' while getting
>>>>> initial credentials
>>>>> [root at ipaclient ~]# nslookup -type=srv _kerberos-master._tcp
>>>>> Server: 172.16.112.8
>>>>> Address: 172.16.112.8#53
>>>>>
>>>>> _kerberos-master._tcp.mpls.local service = 0 100 88
>>>>> ipaserver2.mpls.local.
>>>>> _kerberos-master._tcp.mpls.local service = 0 100 88
>>>>> ipaserver.mpls.local.
>>>>>
>>>>> [root at ipaclient ~]# nslookup -type=srv _kerberos-master._udp
>>>>> Server: 172.16.112.5
>>>>> Address: 172.16.112.5#53
>>>>>
>>>>> _kerberos-master._udp.mpls.local service = 0 100 88
>>>>> ipaserver.mpls.local.
>>>>> _kerberos-master._udp.mpls.local service = 0 100 88
>>>>> ipaserver2.mpls.local.
>>>>>
>>>>>
>>>>> [root at ipaclient ~]# kinit mike
>>>>> kinit: Cannot contact any KDC for realm 'MPLS.LOCAL' while getting
>>>>> initial credentials
>>>>>
>>>>> [root at ipaserver ~]#ifup eth0
>>>>>
>>>>> [root at ipaclient ~]# kinit mike
>>>>> Password for mike at MPLS.LOCAL:
>>>> I'd start with the sssd logs. Is it seeing the main server go offline
>>>> and not switching to the second one? Or is it going into offline mode?
>>>>
>>>> Do you have _srv_ or both servers listed in ipa_server in
>>>> /etc/sssd/sssd.conf?
>>>>
>>>> rob
>>>>
>>> Rob, may be I am missing something but how SSSD is related in this case?
>>> The test is done using kinit not SSSD.
>>>
>>> It would actually be an interesting test to try the same via SSSD for
>>> example do su to mike instead of kinit and see what would happen (watch
>>> SSSD logs with high debug level, 8 for example).
>>> If that works it would probably mean that kinit does not fail over
>>> properly. So this would be a Kerberos kinit bug not IPA/SSSD bug.
>>>
>> AFAIK there is "sssd_krb5_locator_plugin". This plugin changes Kerberos servers dynamically at library level, so kinit should select same server as SSSD.
>>
>> Manual page sssd_krb5_locator_plugin says:
>> If the environment variable SSSD_KRB5_LOCATOR_DEBUG is set to any value debug messages will be sent to stderr.
>>
>> You can execute
>> SSSD_KRB5_LOCATOR_DEBUG=1 kinit ...
> Hello,
>
> [root at ipaclient ~]# SSSD_KRB5_LOCATOR_DEBUG=1 kinit mike
> [sssd_krb5_locator] sssd_krb5_locator_init called
> [sssd_krb5_locator] Found [172.16.112.8] in [/var/lib/sss/pubconf/kdcinfo.MPLS.LOCAL].
> [sssd_krb5_locator] sssd_realm[MPLS.LOCAL] requested realm[MPLS.LOCAL] family[0] socktype[2] locate_service[1]
> [sssd_krb5_locator] addr[172.16.112.8:88] family[2] socktype[2]
> [sssd_krb5_locator] [172.16.112.8] used
> [sssd_krb5_locator] sssd_krb5_locator_close called
> [sssd_krb5_locator] sssd_krb5_locator_init called
> [sssd_krb5_locator] Found [172.16.112.8] in [/var/lib/sss/pubconf/kdcinfo.MPLS.LOCAL].
> [sssd_krb5_locator] sssd_realm[MPLS.LOCAL] requested realm[MPLS.LOCAL] family[0] socktype[1] locate_service[1]
> [sssd_krb5_locator] addr[172.16.112.8:88] family[2] socktype[1]
> [sssd_krb5_locator] [172.16.112.8] used
> [sssd_krb5_locator] sssd_krb5_locator_close called
> [sssd_krb5_locator] sssd_krb5_locator_init called
> [sssd_krb5_locator] Found [172.16.112.8] in [/var/lib/sss/pubconf/kdcinfo.MPLS.LOCAL].
> [sssd_krb5_locator] open failed [2][No such file or directory].
> [sssd_krb5_locator] reading kpasswd address failed, using kdc address.
> [sssd_krb5_locator] sssd_realm[MPLS.LOCAL] requested realm[MPLS.LOCAL] family[2] socktype[2] locate_service[2]
> [sssd_krb5_locator] addr[172.16.112.8:88] family[2] socktype[2]
> [sssd_krb5_locator] [172.16.112.8] used
> [sssd_krb5_locator] sssd_krb5_locator_close called
> Password for mike at MPLS.LOCAL:
> [sssd_krb5_locator] sssd_krb5_locator_init called
> [sssd_krb5_locator] Found [172.16.112.8] in [/var/lib/sss/pubconf/kdcinfo.MPLS.LOCAL].
> [sssd_krb5_locator] open failed [2][No such file or directory].
> [sssd_krb5_locator] reading kpasswd address failed, using kdc address.
> [sssd_krb5_locator] sssd_realm[MPLS.LOCAL] requested realm[MPLS.LOCAL] family[0] socktype[2] locate_service[2]
> [sssd_krb5_locator] addr[172.16.112.8:88] family[2] socktype[2]
> [sssd_krb5_locator] [172.16.112.8] used
> [sssd_krb5_locator] sssd_krb5_locator_close called
> [sssd_krb5_locator] sssd_krb5_locator_init called
> [sssd_krb5_locator] Found [172.16.112.8] in [/var/lib/sss/pubconf/kdcinfo.MPLS.LOCAL].
> [sssd_krb5_locator] open failed [2][No such file or directory].
> [sssd_krb5_locator] reading kpasswd address failed, using kdc address.
> [sssd_krb5_locator] sssd_realm[MPLS.LOCAL] requested realm[MPLS.LOCAL] family[0] socktype[1] locate_service[2]
> [sssd_krb5_locator] addr[172.16.112.8:88] family[2] socktype[1]
> [sssd_krb5_locator] [172.16.112.8] used
> [sssd_krb5_locator] sssd_krb5_locator_close called
>
> [root at ipaserver2 ~]ifdown eth0 # NOTE: ipaserver2 is 172.16.112.8
>
> [root at ipaclient ~]# SSSD_KRB5_LOCATOR_DEBUG=1 kinit mike
> [sssd_krb5_locator] sssd_krb5_locator_init called
> [sssd_krb5_locator] Found [172.16.112.8] in [/var/lib/sss/pubconf/kdcinfo.MPLS.LOCAL].
> [sssd_krb5_locator] sssd_realm[MPLS.LOCAL] requested realm[MPLS.LOCAL] family[0] socktype[2] locate_service[1]
> [sssd_krb5_locator] addr[172.16.112.8:88] family[2] socktype[2]
> [sssd_krb5_locator] [172.16.112.8] used
> [sssd_krb5_locator] sssd_krb5_locator_close called
> [sssd_krb5_locator] sssd_krb5_locator_init called
> [sssd_krb5_locator] Found [172.16.112.8] in [/var/lib/sss/pubconf/kdcinfo.MPLS.LOCAL].
> [sssd_krb5_locator] sssd_realm[MPLS.LOCAL] requested realm[MPLS.LOCAL] family[0] socktype[1] locate_service[1]
> [sssd_krb5_locator] addr[172.16.112.8:88] family[2] socktype[1]
> [sssd_krb5_locator] [172.16.112.8] used
> [sssd_krb5_locator] sssd_krb5_locator_close called
> kinit: Cannot contact any KDC for realm 'MPLS.LOCAL' while getting initial credentials
Jakub, does this make sense to you?
> Thanks,
> Mike
>
>
> _______________________________________________
> Freeipa-users mailing list
> Freeipa-users at redhat.com
> https://www.redhat.com/mailman/listinfo/freeipa-users
--
Thank you,
Dmitri Pal
Sr. Engineering Manager for IdM portfolio
Red Hat Inc.
-------------------------------
Looking to carve out IT costs?
www.redhat.com/carveoutcosts/
More information about the Freeipa-users
mailing list