[Freeipa-devel] Unit tests failing on F18

[Martin Kosek]

On 10/18/2012 12:04 AM, Rob Crittenden wrote:
> Martin Kosek wrote:
>> Hello,
>>
>> I was investigating global unit test failure on Fedora 18 for most of today, I
>> would like to share results I found so far.
>>
>> Unit test and its related scripts on F18 now reports NSS BUSY exception, just
>> like this one:
>>
>> # ./make-testcert
>> Traceback (most recent call last):
>>    File "./make-testcert", line 134, in <module>
>>      sys.exit(makecert(reqdir))
>>    File "./make-testcert", line 111, in makecert
>>      add=True)
>>    File "./make-testcert", line 68, in run
>>      result = self.execute(method, *args, **options)
>>    File "/root/freeipa-master2/ipalib/backend.py", line 146, in execute
>>      raise error #pylint: disable=E0702
>> ipalib.errors.NetworkError: cannot connect to
>> 'http://vm-042.idm.lab.bos.redhat.com/ipa/session/xml': [Errno -8053]
>> (SEC_ERROR_BUSY) NSS could not shutdown. Objects are still in use.
>>
>> Something In F18 must have changed, this worked before... But leaked
>> NSSConnection objects without proper close() now ends with the exception above.
>>
>> In case of make-testcert script, the exception is raised because the script
>> does the following procedure:
>>
>> 1) connect, do one command
>> 2) disconnect
>> 3) connect, do second command
>>
>> However, during disconnect, NSSConnection is leaked which makes NSS very
>> uncomfortable during second connection atempt (and nss_shutdown()). I managed
>> to fix this issue with attached patch. ./make-testcert or "./make-test
>> tests/test_xmlrpc/test_group_plugin.py" works fine now.
>>
>> But global "./make-test" still fails, I think there is some remaining
>> NSSConnection leak, I suspect there is something wrong with how we use our
>> context (threading.local object). It looses a connection or some other thread
>> invoked in ldap2 module may be kicking in, here is my debug output:
>>
>> CONTEXT[xmlclient] = <ipalib.request.Connection object at 0x9a1f5ec>
>>
>> Test a simple LDAP bind using ldap2 ... SKIP: No directory manager password in
>> /root/.ipa/.dmpw
>> Test the `ipaserver.rpcserver.jsonserver.unmarshal` method. ... ok
>> tests.test_ipaserver.test_rpcserver.test_session.test_mount ... CONTEXT
>> 150714476: GET languages
>>
>> CONTEXT[xmlclient] = None
>>
>> The connection is in the context, but then something happens and it is gone.
>> Then, unit tests try to connect again and NSS fails.
>>
>> I would be really glad if somebody with a knowledge of NSS or how threads in
>> Python/IPA work could give me some advice...
>>
>> Thanks!
>> Martin
> 
> I built upon your patch and have something that seems to work at least
> somewhat. I'm getting some unexpected test failures when running the entire
> suite but no NSS shutdown errors. I haven't had a chance to really investigate
> everything yet, sending this out as a work-in-progress in case you want to take
> a look.
> 
> rob
> 

Yeah, this is great! I tested a fresh build+install on Fedora 18 with your
patch and all tests succeeded. So as for F18, I am inclined to ACK the patch as is.

I am just not sure that this will work on platforms with Python version < 2.7,
xmlrpclib is different there.

Martin