Skip to content

Latest commit

 

History

History
143 lines (107 loc) · 5.81 KB

framework.org

File metadata and controls

143 lines (107 loc) · 5.81 KB
Raw

WCT Testing Framework

This describes the WCT test framework.

Overview

WCT provides a large and growing suite of self-tests. When they pass, they assure a variety of correct or at least expected behavior. Examples of the behavior tested:

  • compilation against the WCT API.
  • run-time against the WCT libraries.
  • code functionality at different scales.
  • validity of results from known input.
  • stability of results across releases.

Tests are developed in the WCT testing framework as described in this document.

Executing tests

A user may execute tests in three different ways.

Automated campaign

By default, no tests are executed by the WCT builder (waf or the provided wcb). To execute tests the --tests option may be given at configure time:

waf configure --tests [...]

By default, when tests are enabled, the entire test group sequence is executed in the order listed above. This sequence can be shortened by naming a different terminating group:

waf configure --tests --test-group=atomic [...]

A user may change their mind about the testing. They may either rerun the configure step or they may pass the options again at the waf build phase. For example, waf configure --tests was ran but subsequently the user wants to disable tests or to set a terminating group:

waf --test-group=atomic
waf --notests

Likewise, if testing was not configured (the default), one can turn it back on during building

waf --tests

The values remembered by waf configure may be viewed with:

waf dumpenv|grep 'waf: TEST'

Some tests take many minutes to complete. Each time they are run, their run time is recorded. Subsequent testing campaigns may exclude all test that are expected to require more time than a given test duration:

waf --test-duration=<seconds>

Targeted execution

Assuming the build was configured with --test, an individual test may be run explicitly via waf by giving its “target name”:

waf --target=test_testing

Test target names are derived from source file names with file name extension removed. Interpreted tests will have the interpreter name appended. A listing of all targets including those for tests can be found with:

waf list

To force a particular test to be rerun after it has been run once:

waf --target=test_testing --alltests

If the build was not configured with the --test option, this option must be added to the above examples to enable them to actually operate on tests.

Direct execution

A test program may be executed directly from the shell command line. Compiled tests are found under build/<pkg>/<prefix><sep><name>. Interpreted test files are left in-place in the source trees and must be run with an appropriate interpreter. Here are some examples of directly running tests:

./build/util/test_testing
wcsonnet cfg/test/test_wirecell.jsonnet
bats test/test/test_bats.bats

Test requirements

A test program:

  • should include at least one check for proper behavior that may fail,
  • must exit with a non-zero status code when failure is encountered.
  • must exit with a zero status code when no failure is encountered.
  • should be fast to compile/interpret and execute.
  • should cover a minimum of code scope needed to perform its checks.
  • must not depend on the execution of other tests (in its group).

Test source

The source file for a test program must found as:

<pkg>/test/<prefix><sep><name>.<ext>

With the following elements:

  • <pkg> names the sub package directory name, eg util.
  • <prefix> names a test group or an alias for a group. See below.
  • <sep> is optional and may be an underscore (”_”) or hyphen (”~”).
  • <name> should briefly describe the scope of a test and should follow the uniqueness guidelines given below.
  • <ext> defines how the test should be dispatched (compiled or interpreted).

Some examples:

util/test/test_testing.cxx
gen/test/test-addnoise.bats
util/test/check_act2viz.cxx

Test groups

All tests in a test group are executed independently and each group is executed in sequence. A test in one group may depend on programs or their results from a prior group.

The test groups names are listed. Their <prefix> matches the group name or in some cases aliases are checked.

check
simply compile a program but do not run. The <prefix> is check.
atomic
is a basic test, depending on no other tests (but see Data repository). Besides a <prefix> of atomic the prefix test may be (and usually is) given.
history
intended to run after all atomic tests and potentially consume data from prior campaigns and/or produce data for future campaigns. See Data repository. The <prefix> is history.
report
intended as a final stage to produce material for human inspection. The <prefix> is report.

Finally, the <ext> determines if compilation or interpolation is required. See C++ testing for information about compiled tests.

Most simply, the <name> should made be unique across the entire toolkit. It is however acceptable to have more than one test with a given <name> as long as it differs in <prefix>, <sep> or <ext>. Developers of such tests must assure the tests are mutually compatible. No test may have <prefix>, <sep>, <name> and <ext> in common as only one will be run.