[Freeipa-devel] [PATCH] 1 Do lazy initializiation ipalib
Martin Kosek
mkosek at redhat.com
Wed Nov 2 13:41:43 UTC 2011
On Wed, 2011-11-02 at 12:42 +0200, Alexander Bokovoy wrote:
> On Mon, 31 Oct 2011, Jan Cholasta wrote:
> > Dne 31.10.2011 13:19, Alexander Bokovoy napsal(a):
> > >On Mon, 31 Oct 2011, Jan Cholasta wrote:
> > >>Added finalization for __call__ and the check for CLI. Patch attached.
> > >ACK from my side but see below.
> > >
> > >>+ def __getattribute__(self, name):
> > >>+ if not name.startswith('_Plugin__') and not name.startswith('_ReadOnly__') and name != 'finalize_late':
> > >>+ self.finalize_late()
> > >>+ return object.__getattribute__(self, name)
> > >Could you get faster than three string comparisons? As
> > >__getattribute__ is fairly often called it would make sense to keep
> > >these operations to absolute minimum.
> > >
> >
> > Is there any noticable slowdown?
> Yes. Now I have different patch to solve this issue that avoids using
> __getattribute__. Instead, it sets a trap on attributes that are
> changed by finalization process and when they are accessed first time,
> the trap forces instance to finalize. As result, the attributes get
> their proper values and traps are removed, no performance costs
> anymore.
>
> For Commands one additional check is done on __call__() method to
> verify that we are indeed finalized before execution proceeds. It is a
> safety net here.
>
> Performance is not bad:
>
> 1. Before the patch
> [root at vm-114 ipalib]# time ipa >/dev/null
>
> real 0m1.101s
> user 0m0.930s
> sys 0m0.151s
> [root at vm-114 ipalib]# time ipa user-find>/dev/null
>
> real 0m3.132s
> user 0m0.983s
> sys 0m0.135s
>
> 2. With patch
> [root at vm-114 ipalib]# patch -p2 <~/speedup.patch
> patching file frontend.py
> patching file plugable.py
> [root at vm-114 ipalib]# time ipa >/dev/null
>
> real 0m0.563s
> user 0m0.438s
> sys 0m0.098s
> [root at vm-114 ipalib]# time ipa >/dev/null
>
> real 0m0.521s
> user 0m0.412s
> sys 0m0.100s
> [root at vm-114 ipalib]# time ipa user-find>/dev/null
>
> real 0m1.069s
> user 0m0.445s
> sys 0m0.111s
> [root at vm-114 ipalib]# time ipa user-find>/dev/null
>
> real 0m0.840s
> user 0m0.425s
> sys 0m0.126s
> [root at vm-114 ipalib]# time ipa user-find>/dev/null
>
> real 0m0.816s
> user 0m0.432s
> sys 0m0.119s
>
> Patch is attached.
Hmm, this looks impressive! It would brings us considerably faster CLI
calls.
And again, I would feel much safer if we do these optimizations only
for CLI and let server do all the initialization right on start.
Unit tests show few errors though. But these may not be the actual
problems in the patch, it may be just wrong test assumptions:
======================================================================
FAIL: Doctest: ipalib.crud
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib64/python2.7/doctest.py", line 2166, in runTest
raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for ipalib.crud
File "/home/mkosek/freeipa/ipalib/crud.py", line 19, in crud
----------------------------------------------------------------------
File "/home/mkosek/freeipa/ipalib/crud.py", line 76, in ipalib.crud
Failed example:
list(api.Command.user_add.args)
Exception raised:
Traceback (most recent call last):
File "/usr/lib64/python2.7/doctest.py", line 1254, in __run
compileflags, 1) in test.globs
File "<doctest ipalib.crud[13]>", line 1, in <module>
list(api.Command.user_add.args)
TypeError: 'Finalizer' object is not iterable
----------------------------------------------------------------------
File "/home/mkosek/freeipa/ipalib/crud.py", line 78, in ipalib.crud
Failed example:
list(api.Command.user_add.options)
Exception raised:
Traceback (most recent call last):
File "/usr/lib64/python2.7/doctest.py", line 1254, in __run
compileflags, 1) in test.globs
File "<doctest ipalib.crud[14]>", line 1, in <module>
list(api.Command.user_add.options)
TypeError: 'Finalizer' object is not iterable
----------------------------------------------------------------------
File "/home/mkosek/freeipa/ipalib/crud.py", line 94, in ipalib.crud
Failed example:
list(api.Command.user_show.args)
Exception raised:
Traceback (most recent call last):
File "/usr/lib64/python2.7/doctest.py", line 1254, in __run
compileflags, 1) in test.globs
File "<doctest ipalib.crud[15]>", line 1, in <module>
list(api.Command.user_show.args)
TypeError: 'Finalizer' object is not iterable
----------------------------------------------------------------------
File "/home/mkosek/freeipa/ipalib/crud.py", line 96, in ipalib.crud
Failed example:
list(api.Command.user_show.options)
Exception raised:
Traceback (most recent call last):
File "/usr/lib64/python2.7/doctest.py", line 1254, in __run
compileflags, 1) in test.globs
File "<doctest ipalib.crud[16]>", line 1, in <module>
list(api.Command.user_show.options)
TypeError: 'Finalizer' object is not iterable
----------------------------------------------------------------------
File "/home/mkosek/freeipa/ipalib/crud.py", line 114, in ipalib.crud
Failed example:
list(api.Command.user_add.output_params)
Exception raised:
Traceback (most recent call last):
File "/usr/lib64/python2.7/doctest.py", line 1254, in __run
compileflags, 1) in test.globs
File "<doctest ipalib.crud[18]>", line 1, in <module>
list(api.Command.user_add.output_params)
TypeError: 'NoneType' object is not iterable
----------------------------------------------------------------------
File "/home/mkosek/freeipa/ipalib/crud.py", line 116, in ipalib.crud
Failed example:
list(api.Command.user_show.output_params)
Exception raised:
Traceback (most recent call last):
File "/usr/lib64/python2.7/doctest.py", line 1254, in __run
compileflags, 1) in test.globs
File "<doctest ipalib.crud[19]>", line 1, in <module>
list(api.Command.user_show.output_params)
TypeError: 'NoneType' object is not iterable
======================================================================
FAIL: Test the ``ipalib.frontend.Command.args`` instance attribute.
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/nose/case.py", line 187, in
runTest
self.test(*self.arg)
File "/home/mkosek/freeipa/tests/test_ipalib/test_frontend.py", line
255, in test_args
assert self.cls().args is None
AssertionError
======================================================================
FAIL: Test the `ipalib.frontend.Command.get_output_params` method.
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/nose/case.py", line 187, in
runTest
self.test(*self.arg)
File "/home/mkosek/freeipa/tests/test_ipalib/test_frontend.py", line
710, in test_get_output_params
assert list(inst.get_output_params()) == ['one', 'two', 'three']
AssertionError
======================================================================
FAIL: Test the ``ipalib.frontend.Command.options`` instance attribute.
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/nose/case.py", line 187, in
runTest
self.test(*self.arg)
File "/home/mkosek/freeipa/tests/test_ipalib/test_frontend.py", line
304, in test_options
assert self.cls().options is None
AssertionError
I found one more issue with our ./makeapi script.
$ ./makeapi
Writing API to API.txt
Traceback (most recent call last):
File "./makeapi", line 392, in <module>
sys.exit(main())
File "./makeapi", line 376, in main
rval |= make_api()
File "./makeapi", line 167, in make_api
fd.write('args: %d,%d,%d\n' % (len(cmd.args), len(cmd.options),
len(cmd.output)))
TypeError: object of type 'Finalizer' has no len()
Martin
More information about the Freeipa-devel
mailing list