[Avocado-devel] [RFC] Pre/Post test hooks

Fri Apr 8 10:36:32 UTC 2016

Dne 5.4.2016 v 21:28 Cleber Rosa napsal(a):
>
>
> On 04/05/2016 03:23 PM, Jeff Nelson wrote:
>> On Fri, Apr 01, 2016 at 11:00:39AM -0300, Cleber Rosa wrote:
>>> MOTIVATION
>>> ==========
>>>
>>> The idea of adding hooks to be run by Avocado before and after tests
>>> is general enough, and may be used by the community in unpredictable
>>> ways. And that is good.
>>>
>>> For this team, the initial motivation was to be able to bring back an
>>> Autotest feature that some of our users are missing: the ability to
>>> set the system-wide "kernel core pattern" configuration for tests.
>>>
>>> Having a pre-test hook would allow "/proc/sys/kernel/core_pattern" to
>>> be read, saved and modified to point to the test results directory.
>>> Having a post-test hook would allow "/proc/sys/kernel/core_pattern" to
>>> be reverted back to its original state.
>>>
>>> Other currently core features such as sysinfo collection, could be
>>> re-implemented as pre/post test hooks.
>>>
>>> GENERAL DESIGN POINTS
>>> =====================
>>>
>>> These are the most important design decisions to be acknowledged or
>>> questioned. Please reply with either ACK or your questions/suggestions.
>>
>> I have some questions (hope you don't mind).
>>
>> What are the outputs of pre- and post-test hooks?
>>
>
> The outputs are defined by the actual plugin "hooked in the hook". The
> current interface doesn't define it. They can be completely silent, they
> can generate output to the UI, they can write to files at the test's
> result directory, etc.
>
>> Are there limits to the actions that are permitted in pre- and
>> post-test hooks? Of course, the primary use-case of the pre-test hook
>> is to set up the environment for the test--so environment changes are
>> permitted--and the use-case for a matching post-hook is to restore the
>> environment. About the only operation I can imagine NOT being
>> permitted is to abort (kill itself, or kill its controlling parent
>> process).
>>
>
> The pre/post test hooks are fed the test status, so they can gather
> information, but not (directly) influence its outcome.
>
>> Can a pre-test hook return a status that causes the test execution to
>> be skipped? I can imagine this being done for another use-case:
>> validate the test environment (e.g., check to see if required hardware
>> is present).
>>
>
> No, this should be done at the test's "setUp" stage. Pointers:
>
> https://github.com/avocado-framework/avocado/blob/master/avocado/core/test.py#L368
>
>
> But I understand the value in influencing all tests with a single,
> plugable, block of code. Still, this looks like something that
> could/should be addressed by what we're calling "Job API". With some
> Python code and such an API, you could select the exact tests you want
> to run.
>
>> Can a post-test hook alter the result (status) of the test?
>>
>
> In the current implementation draft it could. But just because we're
> sharing the same (last, final) test state among the post plugins and
> test result handlers. I'm not sure that a plugin *should* change it,
> though.
>
I can imagine a BZ plugin, which marks the results as `SKIP` if assigned 
BZ exists, so maybe it can be useful.

>> Has there been any thought given to having pre- and post-job hooks?
>> For example, setting the kernel core pattern is something I would want
>> to do globally, for all tests in a job. It would be faster and more
>> convenient to do this just once. But I admit this is a +1 optimization
>> and need not be considered now. (It also complicates things when tests
>> run on multiple machines.)
I'm not sure you want to set the kernel core pattern per job. I'd much 
rather see the coredumps inside each test result. (still there is the 
problem with concurrent jobs...)

>
> Pre/Post job hooks are actually simpler, and implemented in the
> following PR:
>
> https://github.com/avocado-framework/avocado/pull/1106
>
>>
>> Can there be multiple hooks for a given test? If so, how does one
>> define the order in which they are executed? Since there are pre- and
>> post-test hooks, there are really two orders to consider.
>>
>
> You mean multiple plugins registered under a given hook, or simply put,
> multiple pre (and/or) post plugins active, right? The answer is yes.
>
> We have not implemented any kind of ordering management, simply because
> we have not hit a use case that requires it.
>
>> I found myself wanting to make an assumption so I better ask: must
>> hooks come in pairs (for every pre-test there must be a post-test, and
>> vice-versa)?
The alternative proposal by me handled this differently. It'd allow you 
to create a `job` plugin, which would be notified of certain situations 
like `pre` and `post` test. So in this version yes, all plugins would 
contain pre and post section, even dummy one.

The benefit would be that if we add `pre_job` and `post_job`, you'd 
control the whole flow inside one plugin (the plugin would be 
initialized once, then it'd be called when the hooks are reached)

The benefit of Cleber's proposal is that each plugin is separated and 
can be registered as pre/post or in the future pre_job, post_job (if we 
decide like adding it). (separate instance of the plugin would be 
initialized for each entry-point - pre, post - and called when the hook 
is reached).

I have one more questions for you, Cleber. How do you plan to manage 
those plugins? Because right now (if I'm not wrong) the only way is to 
install or uninstall the plugin into certain namespace.

Lukáš

>
> Not at all. You can have only pre hooks enabled, only post hooks enabled
> or both.
>
> The given *job* mail notification plugin example, for instance, is
> something that makes more sense activated only in a post hook:
>
> https://github.com/clebergnu/avocado/blob/job_pre_post/examples/plugins/job-pre-post/mail/avocado_job_mail.py
>
>
>>
>> That's all for now.
>>
>> -Jeff
>>
>
> Thanks a lot for the feedback!
> - Cleber.
>
> _______________________________________________
> Avocado-devel mailing list
> Avocado-devel at redhat.com
> https://www.redhat.com/mailman/listinfo/avocado-devel