[Avocado-devel] [RFC] Pre/Post test hooks

[Cleber Rosa]

MOTIVATION
==========

The idea of adding hooks to be run by Avocado before and after tests is 
general enough, and may be used by the community in unpredictable ways. 
And that is good.

For this team, the initial motivation was to be able to bring back an 
Autotest feature that some of our users are missing: the ability to set 
the system-wide "kernel core pattern" configuration for tests.

Having a pre-test hook would allow "/proc/sys/kernel/core_pattern" to be 
read, saved and modified to point to the test results directory. Having 
a post-test hook would allow "/proc/sys/kernel/core_pattern" to be 
reverted back to its original state.

Other currently core features such as sysinfo collection, could be 
re-implemented as pre/post test hooks.

GENERAL DESIGN POINTS
=====================

These are the most important design decisions to be acknowledged or 
questioned. Please reply with either ACK or your questions/suggestions.

1) Hooks are implemented as plugin classes, based on a given defined 
interface, in the same way current "CLICmd" and "CLI" interfaces allow 
plugin writers to extend Avocado and give it new commands and command 
line options.

2) The hooks are executed by the *runner*, and not by the test process. 
The goal is not interfere with the test itself. The pre and post code 
that runs before and after the test should not *directly* change the 
test behavior and outcome. Of course, the test environment can be 
changed in a way (say having packages removed) that a test may fail 
because of hook actions.

3) Test execution time should not be changed by pre and post hooks. If a 
pre-test hook takes "n" seconds to run, "n" should not be added to the 
test run time.

4) Job run time: right now, Avocado times a Job based on the sum of 
individual test run times. With pre and post test hooks, this can be 
very different from job "wall clock" times. My instinct is to change 
that, so that a Job run time is the job "wall clock" time. I'm unsure if 
we should add yet another time measure, that is, the sum of individual 
test run time. This is also bound to be broken when parallel run of 
tests is implemented.

5) The pre test hook is given the test "early status". Information such 
as the test tagged name, the fact that it has not yet started to run and 
the test results directory are all part of the early status.

6) Because of point #5, the test is instantiated on the test process, 
its early state is sent, but the test execution itself is held until the 
runner finishes running the pre-test hooks.

7) The post test hook is given the last test status, which is also used 
by the runner to identify test success, failure, etc.


Thanks,
  - Cleber.

-- 
Cleber Rosa
[ Sr Software Engineer - Virtualization Team - Red Hat ]
[ Avocado Test Framework - avocado-framework.github.io ]