Development Techniques for Using Simulation to Remove Risk in Software/Hardware Integration

Configurable Testing

Once a component developer has taken the decision to develop configurable code, it then becomes necessary to consider how to test this code. In particular, tests must themselves know about what it is they are testing - there is no point testing something that the user has chosen not to include!

For this reason, the same techniques can be applied to testing as were applied to components - test programs should have their own CDL, with their own dependencies. This allows a test only to be run if the configuration is actually applicable to that test.

Similarly, the tests need to react to the configuration being tested, and may need to expect different results depending on that configuration. For that reason, tests must use the same approach as components to deal with the current configuration. In the case of C/C++, this is simply done by the C preprocessor. For example:

#include <pkgconf/libc.h> // C library configuration header file

int main()

{

#ifdef CYGSEM_LIBC_STRTOK_THREAD_SAFE

test_strtok_threadsafety();

#endif

#ifdef CYGIMP_LIBC_STRTOK_FAST

...

Tests have extra dependency problems obviously the functionality being tested must be enabled. But the test may also need other functionality unrelated to what is being tested for the test to run at all. For example, testing thread safety of strtok() may require a kernel C API to be present to be able to set up threads. But that's not a requirement of strtok() itself, which may use an internal API to access thread information.

To the component developer, adding fully configurable tests may seem like an unnecessary overhead, when internal testing of particular configurations may suffice. But as Cygnus has discovered with eCos, distributing these tests to the users along with the components is very beneficial it is impossible for a component developer to test every configuration of the system for all but the most trivial of systems. The user's configuration of the component may be very different from what the component developer tested, and including a test with the component allows the user to tell whether the tested functionality works as expected in their configuration. It may not work due to an interaction the developer had not foreseen.

By providing these fully configurable tests with the component, the user can be more reassured that if there is something going wrong in their system, it is more likely to be their application at fault, rather than the configurable component being used. This helps reduce support costs.

Pre-defined permutation testing
Configurable testing can be combined with automated testing techniques to accelerate and simplify the QA process. To accomplish this, the component developer can predefine certain configurations of their component that should be explicitly tested. These should include the most common configurations of the components, or known boundary conditions where the developer knows that certain parts of the code are only executed in special cases. These would be known to be valid configurations.

Having predefined permutations makes it much easier for the developer to test their code, including checking for regressions, even in the presence of a large number of configuration options. Permutations should also change values for components other than the particular one being tested, to check that there are no unforeseen interactions with the other components.

Sometimes it may be beneficial to choose an invalid configuration as well, to prove that the configuration rules are correct, and that there is indeed a test failure. Similarly, it may be useful to check that functionality that should be disabled is disabled.

Random permutation testing
It is also possible to do random testing of permutations. In particular, this allows testing of the configuration rules, not just the code. It may draw attention to code that depends on external functionality, but that the developer did not anticipate could be configured in a way incompatible with their code.

This can be done trivially with boolean options by enabling and disabling them randomly (but only ever checking valid configurations). The CDL used to configure test programs could also define weightings and biases for the selection of options in order to reflect the more commonly used configurations. This allows extra attention to be paid to these configurations, rather than considering all random configurations equally.

Similarly some numeric values may also be able to be configured. Using CDL, the test developer could list some valid values for numeric options which the testing infrastructure can randomly choose between, again with weightings. It would be a mistake to allow the testing system to choose completely random values for numeric options without an understanding of the effects this is easily provable by considering options that define memory requirements.

The testing techniques described above may sound like they would require significant development overhead. But Cygnus already successfully employs most of these techniques when testing eCos.