getting_started.rst

Getting Started

General notions

is a code coverage analysis tool offering support for a range of coverage metrics and output formats associated with powerful consolidation features letting users assess the combined coverage achievements of multiple program executions.

The set of compilation units for which a user needs to assess coverage is commonly designated as the set of units of interest. This is a central notion to many of the tool's operations, conveyed by the user through command line switches or GPR project file attributes. Units of interest typically include the code under test in a campaign, as opposed to the sources of the test harness infrastructure.

After one or several program executions, the tool computes coverage metrics for a given set of units out of coverage trace data produced by the executions with the assistance of an instrumentation mechanism. The primary mode offered by performs source instrumentation, where the tool produces a modified version of the program sources to keep track of coverage relevant facts along with the program control flow, and output the coverage data when the program terminates. We call source traces the coverage traces produced by programs instrumented this way.

The set of metrics that the tool can assess from source traces corresponds to the set defined by the DO-178B certification standard for civil avionics, that is:

• Statement Coverage, where the tools assesses the coverage status (executed or not) of source statements such as a variable assignment or a subprogram call;
• Decision Coverage, which, in addition to statement coverage, evaluates whether Boolean expressions (decisions in DO178B parlance) have been exercised both True and False, then
• Modified Condition/Decision Coverage, commonly known as MCDC, which requires testing particular variations of individual Boolean operands (conditions in DO178B parlance) within decisions.

All these metrics are defined with respect to source level entities (statements, decisions, conditions), and we commonly refer to such assessments as source coverage analysis. Individual statements, decisions, or conditions within the scope of an analysis are referred to as Source Coverage Obligations or SCOs.

Workflow overview

Figure fig-flow_srctraces depicts the workflow involved in computing coverage based on source instrumentation.

A setup step is first needed to install a tiny runtime library used by the instrumented sources to register coverage facts and output them eventually. This runtime is parameterized for the project at hand, depending on the set of languages involved (Ada, C, C++) and on the kind of target for which the code will be compiled (native, cross with OS, or bareboard). Once the setup is done, users proceed with instrument / build / execute / analyze cycles to produce traces and compute coverage results.

The setup, instrumentation and analysis steps are all driven by a command line tool which exposes a subcommand for each kind of operation; hence for the setup step, to instrument and to compute metrics from traces.

GPR project files and associated command line switches are used to let the tool know about the project sources and to convey the units of interest. Consolidation can be performed by aggregating multiple traces directly, or internal representations of partial coverage results stored in files that we call Coverage Checkpoints.

Source instrumentation based Coverage analysis overview

also provides an alternate workflow that consists in integrating seamlessly into an existing build process possibly not involving the use of gprbuild. This is provided as an experimental feature and is available for C/C++, compiling with gcc/g++ on a linux host. The figure fig-flow_integrated_instr depicts this workflow that is more thoroughly detailed in the section integrated_instr.

Integrated instrumentation based Coverage analysis overview

A simple example

Here we first introduce a very simple example software project structure then demonstrate one basic analysis workflow for this project with actual commands.

The examples provided in most of this manual assume a native configuration and the process is similar cross environments. Typical variations would touch switches controlling where and when coverage trace data is output by the instrumented programs, and default values for such switches are selected by the tool based on --target and --RTS arguments at setup time.

Sample project

We consider an Ada package providing a set of elementary operations over Integer objects, with a spec and body in source files named ops.ads and ops.adb:

-- ops.ads
package Ops is
  type Op_Kind is (Increment, Decrement);

  procedure Apply (Op : Op_Kind; X : in out Integer);
end;

-- ops.adb
package body Ops is
  procedure Apply (Op : Op_Kind; X : in out Integer) is
  begin
     case Op is
        when Increment => X := X + 1;
        when Decrement => X := X - 1;
     end case;
  end;
end;

We will analyze the coverage achieved by the sample unit test driver <Test Driver> below, in test_inc.adb, which exercises the Increment operation only:

-- test_inc.adb
with Ops;
procedure Test_Inc is
  X : Integer := 4;
begin
  Ops.Apply (Ops.Increment, X);
  pragma Assert (X = 5);
end;

Assuming a working directory, with the ops sources in an opslib subdirectory and the test sources in a tests subdirectory, we will use a couple of project files in the common working directory:

-- code.gpr
project Code is
  for Source_Dirs use ("opslib");
  for Object_Dir use "obj-" & Project'Name;
end Code;

-- tests.gpr
with "code.gpr";

project Tests is
  for Source_Dirs use ("tests");
  for Object_Dir use "obj-" & Project'Name;

  for Main use ("test_inc.adb");
end Tests;

Setup, Instrument, Build, Execute, Analyze

The instrumentation step that follows assumes that the original program is well formed. A simple way to verify this is to build the non instrumented version first. For our example, this would be:

gprbuild -f -p -Ptests.gpr

We then first set up the instrumentation context, providing a local prefix location where the runtime and default parameters for future commands are going to be installed:

gnatcov setup --prefix=/path/to/gnatcov-rts

Letting further commands know about the prefix location is achieved by adding <prefix>/share/gpr to the GPR_PROJECT_PATH variable. In a Unix like environment, this would be:

export GPR_PROJECT_PATH=$GPR_PROJECT_PATH:/path/to/gnatcov-rts/share/gpr

This will both let the gprbuild command below locate the gnatcov_rts.gpr project file, and the command find default parameter values.

Instrumentation is performed by a simple command:

gnatcov instrument -Ptests.gpr --level=stmt

The use of tests.gpr, not code.gpr, at this step is important as it lets the instrumenter know about the main subprogram, which needs to be processed specially to dump coverage data.

Building the instrumented program then goes like:

gprbuild -f -p -Ptests.gpr \
   --src-subdirs=gnatcov-instr --implicit-with=gnatcov_rts.gpr

This is the same command as for the regular build, with a couple of additional switches to:

• Instruct the builder to search for the instrumented versions of the sources (--src-subdirs),
• Provide visibility to the builder over the coverage runtime referenced by the instrumented sources (--implicit-with).

Executing the test program in its native environment, as in:

obj-tests/test_inc

then produces a test_inc-<stamp>.srctrace source trace file in the current directory. The -<stamp> suffix is intended to prevent clashes in case of concurrent executions of the program in the same directory. It can be controlled in a variety of ways from the instrumentation command line, documented in the instr-tracename section of this manual.

Analysis of the coverage achieved by previous executions is done with commands. For our example use case, this could for instance be:

gnatcov coverage --level=stmt --annotate=xcov test_inc*.srctrace -Ptests.gpr

Here, we request:

• A source statement coverage assessment with --level=stmt,
• An annotated source report in text format with --annotate=xcov,
• For the complete set of units involved in the executable with -Ptests.gpr.

This produces annotated sources in the project's object directory, with ops.adb.xcov quoted below:

examples/starter/src/ops.adb:
67% of 3 lines covered
Coverage level: stmt
 1 .: package body Ops is
 2 .:    procedure Apply (Op : Op_Kind; X : in out Integer) is
 3 .:    begin
 4 +:       case Op is
 5 +:          when Increment => X := X + 1;
 6 -:          when Decrement => X := X - 1;
 7 .:       end case;
 8 .:    end Apply;
 9 .: end Ops;

The analysis results are visible as + / - annotations on source lines, next to the line numbers. The results we have here indicate proper coverage of all the statements except the one dealing with a Decrement operation, indeed never exercised by our driver.

The command actually also produces reports for ops.ads and test_inc.adb, even though the latter is not really relevant. Focus on specific units can be achieved by providing a more precise set of units of interest at this step, for example by adding --projects=code.gpr to the command line, or setting dedicated attributes in the project files themselves. See the sunits chapter for details on this aspect of the procedure.

Going Further

Each of the steps involved in the process overview presented previously is described in detail in a specific chapter of this manual. The most important ones are:

• src_traces
• sunits
• scov
• consolidation

exemptions is also worth noting here, a mechanism allowing users to define code regions for which coverage violations are expected and legitimate.

The known limitations of the tool are outlined in section instr-limitations.

Conventions used in the rest of this manual

• A number of example commands include a --level=<> switch, which conveys a target coverage criterion when needed. <> is a placeholder for an actual level supported by the tool in this case, such as stmt, stmt+decision, or stmt+mcdc for source criteria.
• Example command lines might also include as <units-of-interest> placeholder, which represents a set of switches conveying the set of units for interest for source coverage assessments. GPR project files provide the most elaborate mechanisms for this purpose and the sunits chapter describes all the available options.