[Freeipa-users] ipa migrate-ds failing when more than 1 namingcontext is available

Fri Jan 27 19:01:37 UTC 2012

On 01/27/2012 07:42 PM, Rob Crittenden wrote:
> Sigbjorn Lie wrote:
>> On 01/27/2012 06:15 PM, Sigbjorn Lie wrote:
>>> On 01/27/2012 03:55 PM, Rob Crittenden wrote:
>>>> Sigbjorn Lie wrote:
>>>>>
>>>>> On Fri, January 27, 2012 15:37, Rob Crittenden wrote:
>>>>>> Stephen Gallagher wrote:
>>>>>>
>>>>>>> On Fri, 2012-01-27 at 15:11 +0100, Sigbjorn Lie wrote:
>>>>>>>
>>>>>>>> Hi
>>>>>>>>
>>>>>>>>
>>>>>>>> The first naming context returned from the LDAP server is always
>>>>>>>> chosen
>>>>>>>> when using migrate-ds. This makes my import fail when I attempt
>>>>>>>> to import users and groups from
>>>>>>>> a previous LDAP server having more than 1 naming contexts 
>>>>>>>> available.
>>>>>>>>
>>>>>>>> The migrate-ds script should accept an option to specify what
>>>>>>>> base_dn I
>>>>>>>> would like to import from.
>>>>>>>>
>>>>>>>> Is there such an option today? I cannot find it...
>>>>>>>>
>>>>>>
>>>>>> Not currently. I noticed this earlier in the week and opened a
>>>>>> ticket on
>>>>>> it, https://fedorahosted.org/freeipa/ticket/2314
>>>>>>
>>>>>>>
>>>>>>> Just to add to this request, if the original LDAP server has a
>>>>>>> defaultNamingContext attribute, it should be honored for
>>>>>>> auto-detecting which base to migrate.
>>>>>>
>>>>>> I'll update the 2314 to ensure we don't forget about this. 389-ds 
>>>>>> just
>>>>>> added support for defaultNamingContext.
>>>>>>
>>>>>
>>>>> Ok, thank you.
>>>>>
>>>>> Anything I can do to work around this issue today? I suppose there
>>>>> is just a file that need to be
>>>>> hacked to set a set a value instead of the auto-detected value... ?
>>>>>
>>>>
>>>> /usr/lib/python*/site-packages/ipalib/plugins/migration.py
>>>>
>>>> ~line 620 you'll see a block starting with the comment "retrieve DS
>>>> base DN".
>>>>
>>>> Comment out the next 8 lines by prefixing them with # (these query to
>>>> get the namingContext then pull the first value out).
>>>>
>>>> Add:
>>>>
>>>> ds_base_dn = 'dc=yourbasedn,dc=com'
>>>>
>>>> Alternatively you could always just add the above line to override
>>>> what is detected. Commenting out just saves an LDAP lookup.
>>>>
>>>> Restart Apache.
>>>
>>>
>>> I already found that file and did that earlier today, however I was
>>> restarting tomcat6, not httpd... my bad. :)
>>>
>>> I have to specify --group-objectclass=posixGroup to get groups
>>> imported, that's fine. But I only get a few users imported. I see that
>>> by default it seem to be looking for objectclass=person. Only a few
>>> user accounts have that objectclass associated, so I add
>>> --user-objectclass=posixAccount as all users have this objectclass
>>> associated with their account.
>>>
>>> $ ipa migrate-ds --user-container='ou=people'
>>> --group-container='ou=group' --bind-dn='cn=directory manager'
>>> --user-objectclass=account --group-objectclass=posixGroup
>>> --schema=RFC2307 --continue ldap://ldapserver:399
>>> ipa: ERROR: an internal error has occurred
>>>
>>> Not good. I look in the /var/log/httpd/error_log file, and I find:
>>>
>>> [Fri Jan 27 18:12:51 2012] [error] ipa: INFO: admin at NONE: ping(): 
>>> SUCCESS
>>> [Fri Jan 27 18:12:52 2012] [error] ipa: ERROR: non-public:
>>> UnicodeDecodeError: 'utf8' codec can't decode byte 0xe5 in position 1:
>>> invalid continuation byte
>>> [Fri Jan 27 18:12:52 2012] [error] Traceback (most recent call last):
>>> [Fri Jan 27 18:12:52 2012] [error] File
>>> "/usr/lib/python2.6/site-packages/ipaserver/rpcserver.py", line 228,
>>> in wsgi_execute
>>> [Fri Jan 27 18:12:52 2012] [error] result = self.Command[name](*args,
>>> **options)
>>> [Fri Jan 27 18:12:52 2012] [error] File
>>> "/usr/lib/python2.6/site-packages/ipalib/frontend.py", line 432, in
>>> __call__
>>> [Fri Jan 27 18:12:52 2012] [error] ret = self.run(*args, **options)
>>> [Fri Jan 27 18:12:52 2012] [error] File
>>> "/usr/lib/python2.6/site-packages/ipalib/frontend.py", line 738, in run
>>> [Fri Jan 27 18:12:52 2012] [error] return self.execute(*args, 
>>> **options)
>>> [Fri Jan 27 18:12:52 2012] [error] File
>>> "/usr/lib/python2.6/site-packages/ipalib/plugins/migration.py", line
>>> 634, in execute
>>> [Fri Jan 27 18:12:52 2012] [error] ldap, config, ds_ldap, ds_base_dn,
>>> options
>>> [Fri Jan 27 18:12:52 2012] [error] File
>>> "/usr/lib/python2.6/site-packages/ipalib/plugins/migration.py", line
>>> 513, in migrate
>>> [Fri Jan 27 18:12:52 2012] [error] search_refs=True # migrated DS may
>>> contain search references
>>> [Fri Jan 27 18:12:52 2012] [error] File
>>> "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 188, in 
>>> new_f
>>> [Fri Jan 27 18:12:52 2012] [error] return f(*new_args, **kwargs)
>>> [Fri Jan 27 18:12:52 2012] [error] File
>>> "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 199, in 
>>> new_f
>>> [Fri Jan 27 18:12:52 2012] [error] return args[0].decode(f(*args,
>>> **kwargs))
>>> [Fri Jan 27 18:12:52 2012] [error] File
>>> "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 139, in 
>>> decode
>>> [Fri Jan 27 18:12:52 2012] [error] return tuple(self.decode(m) for m
>>> in var)
>>> [Fri Jan 27 18:12:52 2012] [error] File
>>> "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 139, in
>>> <genexpr>
>>> [Fri Jan 27 18:12:52 2012] [error] return tuple(self.decode(m) for m
>>> in var)
>>> [Fri Jan 27 18:12:52 2012] [error] File
>>> "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 137, in 
>>> decode
>>> [Fri Jan 27 18:12:52 2012] [error] return [self.decode(m) for m in var]
>>> [Fri Jan 27 18:12:52 2012] [error] File
>>> "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 139, in 
>>> decode
>>> [Fri Jan 27 18:12:52 2012] [error] return tuple(self.decode(m) for m
>>> in var)
>>> [Fri Jan 27 18:12:52 2012] [error] File
>>> "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 139, in
>>> <genexpr>
>>> [Fri Jan 27 18:12:52 2012] [error] return tuple(self.decode(m) for m
>>> in var)
>>> [Fri Jan 27 18:12:52 2012] [error] File
>>> "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 157, in 
>>> decode
>>> [Fri Jan 27 18:12:52 2012] [error] dct[k] = self._decode_dict_val(k, v)
>>> [Fri Jan 27 18:12:52 2012] [error] File
>>> "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 64, in
>>> _decode_dict_val
>>> [Fri Jan 27 18:12:52 2012] [error] return self.decode(val)
>>> [Fri Jan 27 18:12:52 2012] [error] File
>>> "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 137, in 
>>> decode
>>> [Fri Jan 27 18:12:52 2012] [error] return [self.decode(m) for m in var]
>>> [Fri Jan 27 18:12:52 2012] [error] File
>>> "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 132, in 
>>> decode
>>> [Fri Jan 27 18:12:52 2012] [error]
>>> var.decode(self.encoder_settings.decode_from)
>>> [Fri Jan 27 18:12:52 2012] [error] File
>>> "/usr/lib64/python2.6/encodings/utf_8.py", line 16, in decode
>>> [Fri Jan 27 18:12:52 2012] [error] return codecs.utf_8_decode(input,
>>> errors, True)
>>> [Fri Jan 27 18:12:52 2012] [error] UnicodeDecodeError: 'utf8' codec
>>> can't decode byte 0xe5 in position 1: invalid continuation byte
>>> [Fri Jan 27 18:12:52 2012] [error] ipa: INFO: admin at NONE:
>>> migrate_ds(u'ldap://svg-p-idm02.none:389', u'********',
>>> binddn=u'cn=directory manager', usercontainer=u'ou=people',
>>> groupcontainer=u'ou=group', userobjectclass=(u'account',),
>>> groupobjectclass=(u'posixGroup',), userignoreobjectclass=None,
>>> userignoreattribute=None, groupignoreobjectclass=None,
>>> groupignoreattribute=None, groupoverwritegid=False, schema=u'RFC2307',
>>> continue=False, exclude_groups=None, exclude_users=None):
>>> UnicodeDecodeError
>>>
>>>
>>> Any suggestions?
>>>
>>>
>> Oh yes and of course I've already looked for accounts with any non-utf8
>> chars in any of the output of an ldapsearch of the same ldap tree I'm
>> trying to import from...
>
> This came up yesterday internally too. I don't believe a bug or ticket 
> has been filed yet.
>
> My best guess on what is happening, based on what I saw with our own 
> case, is this:
>
> A migrated attribute is coming in that IPA doesn't know about. We 
> default to treating all attributes that don't require special 
> treatment (like binary values) as utf-8. I'm guessing that an unknown 
> binary attribute is being migrated, we try to utf-8 encode it and 
> kablooey.
>
> There is no easy workaround for the above problem right now because 
> that happens before the --ignore* options are considered.
>
> One idea that has come to mind but is untested is to grab the missing 
> schema in the form of a discrete 389-ds-compatible schema file. Stop 
> dirsrv, drop the file into /etc/dirsrv/slapd-YOURINSTANCE/schema, 
> start dirsrv
>
> Then retry the migration.
>
> Use --ignore* if you don't want/need these attributes, then you stop 
> dirsrv, remove the schema file, start dirsrv.
>

I did use the ignore flags, ended up with: 
--user-ignore-attribute=memberOf,mail,telephoneNumber,userPassword 
--user-ignore-objectclass=organizationalPerson,posixAccount,shadowAccount,inetOrgPerson,organizationalPerson

Still no luck, same error.

According to your testing: is there still some value that should have 
been ignored I've overlooked ?

On the other hand, I've only been looking at entries within the ou's 
that contains users and groups. Will this issue also occur if there is 
non-UTF8 chars in other parts of the directory?

Regards,
Siggi