froglogic logo

Code Coverage Levels

Squish Coco supports the following code coverage levels:

Each of the code coverage levels provides different metrics some of which are requirements for safety standards and mission critical software.

Function coverage

Code Coverage LevelsWith this metric, Squish Coco counts which functions were called and how often. “Functions” includes also the member functions (or methods) in object-oriented programming languages like C++.

While this metric does not provide many details it can nevertheless be useful for an initial assessment of a project’s test coverage status.

IEC 61508 highly recommends 100% structural test coverage of entry points (i.e. functions) for SIL 1, 2, 3 and 4.

Line Coverage

Here the number of executed code lines is counted and how often each one of them was executed. Only lines that contain executable statements are instrumented, not those with pure declarations. The coverage of a program is then the number of executed lines divided by the number of instrumented lines.

Line coverage is however an unstable coverage measure since it depends strongly on the way the code is formatted.

Statement (Block) Coverage

With this metric, the program statements being executed are tracked. The statement coverage of a program is the number of executed statements in it, divided by the total number of statements. A block is a grouped sequence of statements that can be treated like a single statement. In the C-family of languages curly braces { and } are used for grouping.

A statement block with no jump instructions is also called basic block. Given a single entry and a single exit point only Statement Coverage can be simplified by basing calculations on complete blocks. Unlike Line Coverage this metric is robust against plain code formatting changes. At the same time 100% Statement Coverage implies 100% Line Coverage.

ISO 26262 highly recommends Statement Coverage for ASIL A and B. ASIL C and D recommend it as well but the more strict Branch Coverage and Modified Condition/Decision Coverage are highly recommended instead.

EN 50128 recommends Statement Coverage for SIL 0.

IEC 61508 recommends Statement Coverage for SIL 1 and highly recommends this level for SIL 2, 3 and 4.

Decision Coverage (or Branch Coverage)

Here Squish Coco verifies that all statements are executed and all decisions have all possible results. The coverage of a program is the number of executed statement blocks and decisions divided by the total number of statements and decisions – where each decision counts twice, once for the true case and one for the false case.

ISO 26262 recommends Branch Coverage for ASIL A and highly recommends the method for ASIL B, C and D.

EN 50128 recommends Branch Coverage for SIL 1 and 2. For SIL 3 and 4 this level is even highly recommended.

DO 178C mandates that Decision Coverage should be satisfied with independence for Software Level A and B.

IEC 61508 recommends Branch Coverage for SIL 1 and 2 and highly recommends this level for SIL 3 and 4.

Condition Coverage

This is like Decision Coverage, except that the decisions are split into elementary subexpressions (or conditions) that are connected by AND or OR operators. The coverage of a program is the number of executed statement blocks and conditions divided by the total number of them. Here each condition counts twice, which may result in a large number of possible outcomes in a complex decision.

MC/DC – Modified Condition/Decision Coverage

Modified Condition/Decision Coverage (MC/DC) is a code coverage metric that is referred to in many safety standards as the highest level of quality. It was first defined to provide an intermediate metric between Condition and Multiple Condition Coverage for integrated circuits. Whereas plain condition coverage was considered too simple, the Multiple Condition Coverage has the drawback of a potential exponential explosion of test cases.

MC/DC is a compromise that guarantees a number of test cases that merely grows linearly with the number of conditions, while still achieving a higher level of confidence compared to basic Condition Coverage.

The DO-178C standard defines MC/DC thus:

Every point of entry and exit in the program has been invoked at least once, every condition in a decision in the program has taken all possible outcomes at least once, every decision in the program has taken all possible outcomes at least once, and each condition in a decision has been shown to independently affect that decision’s outcome.

A condition is shown to independently affect a decision’s outcome by: (1) varying just that condition while holding fixed all other possible conditions or (2) varying just that condition while holding fixed all other possible conditions that could affect the outcome.

The sentence in bold was added on top of the definition in DO-178B and precisely permits the behaviour of a short circuit. The short circuit (also called passive evaluation) does not exist in integrated circuits because it corresponds to a software optimisation which consists of skipping the evaluation of some boolean expressions which do not influence the decision outcome. This is why the extension of this definition was not present in earlier versions of the standard.

See a worked example of the test cases required by MC/DC for a small sample of source code.

ISO 26262 recommends MC/DC for ASIL A, B, C and highly recommends this method for ASIL D.

EN 50128 recommends MC/DC (or Multiple Condition Coverage) for SIL 1 and 2. For SIL 3 and 4 this level is even highly recommended.

DO 178C mandates that MC/DC should be satisfied with independence for Software Level A.

IEC 61508 recommends MC/DC for SIL 1, 2 and 3 and highly recommends this level for SIL 4.

MCC – Multiple Condition Coverage

In this coverage metric, also known as Multicondition Coverage and Condition Combination Coverage, all statements must be executed and all combinations of truth values in each decision must occur at least once to reach full coverage.

The coverage of a program is the number of executed statement blocks and condition combinations divided by their total number in the program. MCC has a drawback in so far as the number of possible combinations to be tested can be enormous if the software contains large numbers of conditions. To mitigate this problem the Modified Condition/Decision Coverage metric was created.

100% Multiple Condition Coverage implies 100% Modified Condition/Decision Coverage, 100% Condition Coverage, etc.

See a worked example of the test cases required by MCC for a small sample of source code.

EN 50128 recommends MCC (or Modified Condition/Decision Coverage) for SIL 1 and 2. For SIL 3 and 4 this level is even highly recommended.

About froglogic

froglogic was founded to create a best-of-breed cross-platform test automation tools. The froglogic Squish Testing Suite consists of the cross-platform and multi-language code coverage analysis tool Squish Coco and the GUI Test Automation Tool Squish GUI Tester. More details…

Find out more…

For more information on which code coverage levels are most appropriate for your application and how Squish Coco can generate the metrics you require please complete the form below.

Full Name (required):

Company Name (required):

Your Email (required)