[Freeipa-users] Fw: ssh problem with migrated FreeIPA client on EL7.1 -->Solved
Christopher Lamb
christopher.lamb at ch.ibm.com
Thu Jun 4 17:34:05 UTC 2015
Hi All
I can now report back success (at least on my throwaway EL7.1 test VM).
To switch an EL 7.1 + ipa-client 4.1 host from an old FreeIPA 3.3.3 KDC to
a new FreeIPA 4.1 KDC 3 steps are required:
1) ipa-client-install --uninstall
2) rm -f /var/lib/sss/db/*
3) ipa-client-install --server ldap.my.example.com --domain my.example.com
-N
Having done this, my free-ipa user successfully authenticates (e.g. ssh
remote login with free-ipa user / password
To switch EL 6.5 + ipa-client 3.3.3 hosts step 2) was not required.
Kudos and thanks go to Rob C for suggesting step 2. (Note that the
directory to be purged is /var/lib/sss/db/, not /var/lib/sssd/db/ as
suggested earlier in this thread.
Cheers
Chris
From: Martin Kosek <mkosek at redhat.com>
To: Christopher Lamb/Switzerland/IBM at IBMCH,
freeipa-users at redhat.com
Cc: Jakub Hrozek <jhrozek at redhat.com>, Rob Crittenden
<rcritten at redhat.com>
Date: 03.06.2015 10:39
Subject: Re: [Freeipa-users] Fw: ssh problem with migrated FreeIPA
client on EL7.1 -->Not Solved
On 06/03/2015 10:30 AM, Christopher Lamb wrote:
> Hi all
>
> This is a quick(ish) note to bring everybody up to speed on this issue.
> Yesterday we had some private mail exchange on this issue as I did not
wish
> to broadcast the krb5 and ipa install logs to the user list.
>
> The basic situation is that we are in the process of migrating from an
> FreeIPA 3.3.3 Server (KDC) to a new FreeIPA 4.1 Server (KDC). As
discussed
> in a thread some weeks ago we did not do this by replicating (as perhaps
we
> should have done). Instead we migrated the users across.
>
> We have 30+ servers that are IPA clients ("Hosts" in ipa-speak) joined to
> the old KDC. We are now in the process of migrating these hosts to the
new
> 4.1 KDC.
>
> Most of the hosts run EL 6.5 + ipa-client 3.3.3. For all of these
joining
> to the new KDC was trouble free, taking a few minutes each. After joining
> the new KDC FreeIPA users authenticated properly.
>
> We also had a small number of new EL 7.1 + ipa-client 4.1 hosts that were
> joined direct to the new 4.1 KDC, never having been joined of the 3.3.3
> KDC. These were also trouble free.
>
> The problem occurs with a handful of existing EL 7.1 +ipa-client 4.1
hosts
> that were originally joined to the 3.3.3 KDC, and must be moved to join
the
> 4.1 KDC. These machines no longer authenticate valid FreeIPA users. I
have
> been able to reproduce this behaviour with a freshly setup VM joined
first
> to the 3.3.3 KDC, then moved to the 4.1 KDC.
>
> While the errors show in the krb5 child logs indicate that the password
is
> incorrect, the same user / password is happily accepted by all the other
> hosts.
>
> It seems that in the process of moving / migrating the EL 7.1 /
ipa-client
> 4.1 from the old KDC to the new KDC, "something" is left behind that
causes
> problems. We have seen indications in the install logs that the kinit
steps
> called during ipa-client install are getting responses from the wrong
(old)
> KDC, and not from the new KDC.
>
> Frustratingly. over the weekend i managed to get one of the problem EL
7.1
> boxes to work. However I can't work out exactly what I was that I did
that
> did the trick. However it seems that some kind of major de-install /
> cleanup + reinstall of the ipa-client may be needed.
>
> Rob has suggested that as part of such a cleanup I should do "rm
> -f /var/lib/sssd/db/*". I will test this later today and report back.
>
> Thanks to Rob, Jakub, Martin, Alexander et al for their help and
> suggestions so far.
>
> Chris
Thanks for the background. The pain you are getting is exactly the reason
why
migration via replication to RHEL-7.1 is a better choice :-) Please let us
know
the result, I am curious how this works out.
>
>
>
>
> From: Martin Kosek <mkosek at redhat.com>
> To: Christopher Lamb/Switzerland/IBM at IBMCH,
> freeipa-users at redhat.com, Jakub Hrozek <jhrozek at redhat.com>
> Date: 03.06.2015 09:34
> Subject: Re: [Freeipa-users] Fw: ssh problem with migrated
FreeIPA
> client on EL7.1 -->Not Solved
>
>
>
> On 06/02/2015 06:15 PM, Christopher Lamb wrote:
>>
>> Hi
>>
>> Earlier today I setup 2 throwaway EL7.1 VMs to help narrow down the
cause
>> of this problem. Let's call them HOST09 and HOST10
>>
>> Both are mimimum installs of EL7.1, with NTPD installed and configured.
>>
>> HOST09 had ipa-client 4.1 installed via yum, and was configured to use
> our
>> new FreeIPA 4.1 server, right from the start. --> My FreeIPA user
>> authenticates successfully against this machine.
>>
>> HOST10 had ipa-client 4.1 installed as a dependency of one of our
> standard
>> config packages, and was first set to use our old FreeIPA 3.3.3 server.
> -->
>> My FreeIPA user authenticates successfully. against this machine.
>>
>> I then de-registered HOST10 from the FreeIPA 3.1 server, and registered
>> against the new FreeIPA 4.1 server --> My FreeIPA users does NOT
>> authenticate successfully.
>>
>> This replicates well the behaviour I saw with my production servers,
> namely
>> a) EL 7.1 hosts with ipa-client 4.1 registered directly against the new
> 4.1
>> FreeIPA server authenticate properly.
>>
>> b) EL 7.1 hosts with ipa-client 4.1 first registered against the old
> 3.3.3
>> FreeIPA server, then reregistered with the new 4.1 FreeIPA server do NOT
>> authenticate properly
>>
>> Chris
>
> Hello,
>
> This is really strange. What I do not fully understand is what is the
> "registration against a FreeIPA server". What server you install IPA
client
> should matter if the deployment is set up properly. The host enrollment
> entry
> should simply replicate to whole infrastructure. The only thing that will
> probably differ is sssd.conf and krb5.conf as they will have different
> primary
> server set up, based on what your DNS setup is.
>
> It rather seems that the "reregistration" is what causes the issue. It
> looks
> like something cleanup problem during the process. I will let Jakub to
help
> here, I would suggest including the SSSD logs from the failed login, it
may
> help.
>
>>
>>
>>
>> ----- Forwarded by Christopher Lamb/Switzerland/IBM on 02.06.2015 16:52
>> -----
>>
>> From: Christopher Lamb/Switzerland/IBM at IBMCH
>> To: Jakub Hrozek <jhrozek at redhat.com>
>> Cc: freeipa-users at redhat.com
>> Date: 02.06.2015 10:40
>> Subject: Re: [Freeipa-users] Fw: ssh problem with
migrated
> FreeIPA
>> client on EL7.1 -->Not Solved
>> Sent by: freeipa-users-bounces at redhat.com
>>
>>
>>
>> Hi Jakub
>>
>> Yes root login works, that's how I've been getting into the box.
>>
>> Surprisingly, kinit with my user seems to work on that box. After
> entering
>> my password when prompted, it returns to the commandline without error.
>>
>> However if I try kinit with another FreeIPA user, then instead of
> prompting
>> for a password, it gives "Generic preauthentication failure while
getting
>> initial credentials" error.
>>
>> Having set debug_level=10, when I try and ssh in with my FreeIPA user, I
>> find errors like
>>
>> "Retrieving host .... with result: .. Matching credential not found"
>>
>> "Received error from KDC ... Additional pre-authentication required"
>>
>> "Received error from KDC... Decrypt integrity check failed"
>>
>> "Received error code 1432158219"
>>
>> Cheers
>>
>> Chris
>>
>>
>>
>>
>>
>> From: Jakub Hrozek
<jhrozek at redhat.com>
>> To: Christopher
Lamb/Switzerland/IBM at IBMCH
>> Cc:
freeipa-users at redhat.com
>> Date: 02.06.2015 09:50
>> Subject: Re:
[Freeipa-users] Fw: ssh problem with
> migrated
>> FreeIPA
>> client on EL7.1 -->Not Solved
>>
>>
>>
>> On Tue, Jun 02, 2015 at 09:43:48AM +0200, Christopher Lamb wrote:
>>> Hi Jakub
>>>
>>> The same user / password works with all our FreeIPA hosts - just this
> one
>>> box is the problem. So the password should be good. Of course a type is
>>> always possible (especially for strong passwords), but I have tried
many
>>> times which should eliminate the odd password typo. The user / password
>>> should also be good for both the old and the new FreeIPA Server.
>>
>> Interesting, can you add debug_level=10 to the domain section of
>> sssd.conf? Then krb5_child.log should show Kerberos tracing info
>> including which exact KDC SSSD was talking to.
>>
>>>
>>> As I can neither log in direct, or via ssh to this box with my FreeIPA
>>> user, I assume Kinit with my user won't work- i will try later in the
>> day.
>>
>> Well, login as a UNIX user (root) should work..
>>
>>>
>>> My working assumption is that the problem is related in some way to the
>>> fact the host originally was a FreeIPA 3.3.3 client, updated to FreeIPA
>>> 4.1, and switched between 2 FreeIPA servers. I am currently setting up
2
>>> throwaway EL 7.1 VMs to better test this. On one I will first install
>>> 3.3.3, then upgrade to 4.1. The second will have a direct install of
4.1
>>> client.
>>>
>>> Cheers
>>>
>>> Chris
>>>
>>>
>>>
>>> From:
Jakub Hrozek
> <jhrozek at redhat.com>
>>> To:
> freeipa-users at redhat.com
>>> Date:
02.06.2015 09:22
>>> Subject:
Re:
> [Freeipa-users] Fw: ssh problem with
>> migrated
>> FreeIPA
>>> client on EL7.1 -->Not Solved
>>> Sent by:
> freeipa-users-bounces at redhat.com
>>>
>>>
>>>
>>> On Mon, Jun 01, 2015 at 07:35:11PM +0200, Christopher Lamb wrote:
>>>>
>>>> Hi All
>>>>
>>>> Bad news.
>>>>
>>>> Over the weekend I was able to get the original problem EL7.1 /
FreeIPA
>>> 4.1
>>>> host (FreeIPA client) to authenticate FreeiPA users (my test being ssh
>>>> remote login with FreeIPA user and password).
>>>>
>>>> Today I tried a second machine, and had the same problem, ssh
>> connections
>>>> with FreeIPA user cause "[sssd[krb5_child[3445]]]: Decrypt integrity
>>> check
>>>> failed"
>>>
>>> This really just means wrong password, can you kinit as that user using
>>> the same password?
>>>
>>>>
>>>> Ahh I thought, I have a solution for that: just remove ipa-client and
>>>> reinstall via yum, register with the new FreeIPA server ....
>>>>
>>>> Only with this second machine I still can't ssh in with a FreeIPA
user.
>>>> Argg.....
>>>>
>>>> b.t.w, as this machine is a real physical server, I was able to try
>>> logging
>>>> in direct with my FreeIPA user --> "Authentication Failure"
>>>>
>>>> I now have
>>>> * a whole bunch of EL6.5 / FreeIPA 3.3.3 hosts that migrated from the
>> old
>>>> FreeIPA server to the new without a hitch (i.e. they successfully
>>>> authenticate FreeIPA users.)
>>>> * one migrated EL7.1 / FreeIPA 4.1 host that I was able to migrate,
but
>>>> with problems
>>>> * one migrated EL7.1 / FreeIPA 4.1 host that so far defies all
attempts
>>> to
>>>> authenticate with a FreeIPA user
>>>> * one EL7.1 / FreeIPA 4.1 host that was only ever registered with the
>> new
>>>> FreeIPA server, and successfully authenticates FreeIPA users.
>>>>
>>>> Any ideas?
>>>>
>>>> Chris
>>>>
>>>>
>>>> ----- Forwarded by Christopher Lamb/Switzerland/IBM on 01.06.2015
19:17
>>>> -----
>>>>
>>>> From:
> Christopher
>> Lamb/Switzerland/IBM at IBMCH
>>>> To:
> Alexander Bokovoy
>> <abokovoy at redhat.com>,
>>>> freeipa-users at redhat.com
>>>> Date:
> 30.05.2015 18:52
>>>> Subject:
> Re:
>> [Freeipa-users] ssh problem with
>> migrated FreeIPA
>>> client on
>>>> EL7.1 --> Solved
>>>> Sent by:
>> freeipa-users-bounces at redhat.com
>>>>
>>>>
>>>>
>>>> Hi All
>>>>
>>>> It gives me pleasure to report the problem is solved - a minute ago I
>> was
>>>> able to login via ssh with my FreeIPA user to the problem server,
while
>>>> sitting on my terrace with a glass of wine!
>>>>
>>>> Thanks to Alexander for his helpful advice - we had some mail exchange
>>>> outside the user list as I did not wish to broadcast content of keys,
>>>> config files etc.
>>>>
>>>> Regardless of what I did with commands like klist, kvno everything
>> seemed
>>>> "ok", but I still could not ssh in. Even a ipa-getkeytab did not help.
>>>>
>>>> Therefore I decided to opt for brute force and (partial) ignorance. I
>>>> completely uninstalled the FreeIPA client, and then reinstalled,
>>> configured
>>>> - ét voilà I could ssh in!
>>>>
>>>> This leaves the enigma: what caused the problem? I suspect the
>> following:
>>>>
>>>> The host is an EL 7.1, but the first FreeIPA client installed was
>> version
>>>> 3.3.3 (installed as set of standard packages that we bung on all our
>>>> servers).
>>>>
>>>> This worked fine to authenticate against our "old" 3.x FreeIPA server,
>>> but
>>>> did not work against the "new" 4.1 FreeIPA Server.
>>>>
>>>> When I realised I could not ssh in, one of the first things I did was
>> to
>>>> yum update the FreeIPA client from 3.3.3 to 4.1 - but that did not
>> help.
>>>> The solution was to yum remove the FreeIPA client, then yum install
the
>>> 4.1
>>>> client.
>>>>
>>>> I have some more EL 7.1 servers with the FreeIPA 3.3.3 client
>> installed,
>>> so
>>>> it will be interesting to see it the problem can be reproduced.
>>>>
>>>> Keep up the good work,
>>>>
>>>> Chris
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> From:
>>
Alexander Bokovoy
>> <abokovoy at redhat.com>
>>>> To:
>>
Christopher
>> Lamb/Switzerland/IBM at IBMCH
>>>> Cc:
>> freeipa-users at redhat.com
>>>> Date:
>>
29.05.2015 18:04
>>>> Subject:
>>
Re:
>> [Freeipa-users] ssh problem with
>>> migrated FreeIPA
>>>> client on
>>>> EL7.1
>>>>
>>>>
>>>>
>>>> On Fri, 29 May 2015, Christopher Lamb wrote:
>>>>>
>>>>> Hi All
>>>>>
>>>>> Some weeks ago I setup a new FreeIPA 4.1.0 on an OEL 7.1 server to
>>> replace
>>>>> the existing FreeIPA 3.0.0 running on OEL 6.5, and successfully
>> migrated
>>>>> across the users.
>>>>>
>>>>> We have 50 odd Servers that are FreeIPA clients. Today I started
>>> migrating
>>>>> these one-by-one from the old FreeIPA 3.x server to the new FreeIPA 4
>>>>> server by doing an ipa-client-install --uninstall from the old, and
>>>>> ipa-client-install to register with the new 4.1.0 server.
>>>>>
>>>>> Most of the FreeIPA clients are running OEL 6.5, and for these the
>>>>> migration process above worked perfectly. After migrating the server,
>> I
>>>>> could ssh in with my FreeIPA user.
>>>>>
>>>>> Then I migrated an OEL 7.1 server. The migration itself seemed to
>> work,
>>>> and
>>>>> getent passwd was successful for my FreeIPA user. However when I try
>> and
>>>>> ssh in, my FreeIPA user / password is not accepted.
>>>>>
>>>>> Before the migration I could ssh into the problem server (though
>>> evidently
>>>>> it was using my FreeIPA user from the old FreeIPA server).
>>>>>
>>>>> I can ssh in with a local (non ldap) user, so ssh is running and
>>> working.
>>>>>
>>>>> >From user root I can successfully su to my FreeIPA user.
>>>>>
>>>>> Further investigation showed that version of ipa-client installed was
>>>>> 3.3.3, so I yum updated this to 4.1.0.
>>>>>
>>>>> However I still cannot ssh into the OEL 7.1 box with my FreeIPA user.
>>> The
>>>>> same user continues to work for the 6.5 boxes.
>>>>>
>>>>> A colleague tried to ssh in with his FreeIPA user, and was also
>>> rejected,
>>>>> so the problem is not my user, but is probably for all FreeIPA users.
>>>>>
>>>>> A failed ssh login attempt causes the following error
>>> in /var/log/messages
>>>>>
>>>>> [sssd[krb5_child[5393]]]: Decrypt integrity check failed
>>>> It means /etc/krb5.keytab contains keys from older system and SSSD
>>>> picks them up.
>>>> Can you show output of 'klist -kKet'?
>>>> --
>>>> / Alexander Bokovoy
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Manage your subscription for the Freeipa-users mailing list:
>>>> https://www.redhat.com/mailman/listinfo/freeipa-users
>>>> Go to http://freeipa.org for more info on the project
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Manage your subscription for the Freeipa-users mailing list:
>>>> https://www.redhat.com/mailman/listinfo/freeipa-users
>>>> Go to http://freeipa.org for more info on the project
>>>
>>> --
>>> Manage your subscription for the Freeipa-users mailing list:
>>> https://www.redhat.com/mailman/listinfo/freeipa-users
>>> Go to http://freeipa.org for more info on the project
>>>
>>>
>>>
>>>
>>
>>
>>
>>
>>
>> --
>> Manage your subscription for the Freeipa-users mailing list:
>> https://www.redhat.com/mailman/listinfo/freeipa-users
>> Go to http://freeipa.org for more info on the project
>>
>>
>>
>>
>
>
>
>
More information about the Freeipa-users
mailing list