bannerc
SWE-189 - Code Coverage Measurements

1. Requirements

4.5.9 The project manager shall ensure that the code coverage measurements for the software are selected, implemented, tracked, recorded, and reported.

1.1 Notes

This requirement can be met by running unit, integration, and validation tests; measuring the code coverage; and achieving the code coverage by additional (requirement based) tests, inspection, or analysis. If the project does not get 100 percent structural coverage, it means one of four things and each requires action on the project manager’s part: requirement missing

- the code that hasn’t been covered is performing an essential activity, but no requirement indicates that this should be done; test missing

- the code that hasn’t been covered relates to an existing requirement, but no test was implemented for it; extraneous/dead code

– the code that hasn’t been covered is not traceable to any requirement and isn’t needed by the software; deactivated code

- the code that hasn’t been covered isn’t traceable to any requirements for the current system, but is intended to be executed in specific configurations. The code coverage data and any rationale for uncovered code should be presented and reviewed at major project milestones.

1.2 History

Click here to view the history of this requirement: SWE-189 History

1.3 Applicability Across Classes

 

Class

     A      

     B      

     C      

     D      

     E      

     F      

Applicable?

   

   

   

   

   

   

Key:    - Applicable | - Not Applicable
A & B = Always Safety Critical; C & D = Sometimes Safety Critical; E - F = Never Safety Critical.

2. Rationale

To identify which lines of source code have been tested and which lines of your source code have not been tested and to provide data showing the completeness of the executed software tests.

3. Guidance

Code coverage is an important aspect of good engineering practice. Code coverage is a software testing metric that shows you which lines of your source code have been tested and which lines of your source code have not been tested. This insight allows you to improve your testing so you can improve your code. Code coverage shows which areas of code have been tested ("covered") and which have not. Code coverage provides critical information to show teams where to focus their testing.

Code coverage measurement simply determines which statements in a body of code have been executed through a test run, and which statements have not.  Code coverage measurements for the software should be selected, implemented, tracked, recorded, and reported.

In summary, code coverage is measured for the following reasons:

  • To know how well our tests test our code
  • To know whether we have enough testing in place
  • To maintain the test quality over the lifecycle of a project

If the project does not get 100% structural coverage, it means one of four things and each requires action on the project manager’s part:

  • Requirement missing - the code that hasn’t been covered is performing an essential activity, but no requirement indicates that this should be done
  • Test missing - the code that hasn’t been covered relates to an existing requirement, but no test was implemented for it
  • Extraneous/dead code - The code that hasn’t been covered is not traceable to any requirement and isn’t needed by the software
  • Deactivated code - the code that hasn’t been covered isn’t traceable to any requirements for the current system but is intended to be executed in certain configurations.

To measure what percentage of code has been exercised by a test suite, one or more coverage criteria are used. Coverage criteria are usually defined as rules or requirements, which a test suite needs to satisfy.

There are several coverage criteria, the main ones being:

  • Function coverage – Has each function (or subroutine) in the program been called?
  • Statement coverage – Has each statement in the program been executed?
  • Branch coverage – Has each branch of each control structure (such as in if and case statements) been executed? For example, given an, if statement, have both the true and false branches been executed? Another way of saying this is, has every edge in the program been executed?
  • Condition coverage (or predicate coverage) – Has each Boolean sub-expression evaluated both to true and false?

 Below is a suggestion on code coverage for software classes.

Code coverage versus Software Classification

A

B

C

D

Source code statement coverage

100%

100%

AM

AM

Source code decision coverage

100%

100%

AM

AM

Source code modified condition and decision coverage

100%

AM

AM

AM

NOTE: “AM“ means that the value is agreed and signed off by the Center’s Engineering TA and measured.

Additional guidance related to code coverage may be found in the following related requirements in this Handbook:

4. Small Projects

No additional guidance is available for small projects.

5. Resources

5.1 References

No references have been currently identified for this SWE. If you wish to suggest a reference, please leave a comment below.


5.2 Tools

Tools to aid in compliance with this SWE, if any, may be found in the Tools Library in the NASA Engineering Network (NEN). 

NASA users find this in the Tools Library in the Software Processes Across NASA (SPAN) site of the Software Engineering Community in NEN. 

The list is informational only and does not represent an “approved tool list”, nor does it represent an endorsement of any particular tool.  The purpose is to provide examples of tools being used across the Agency and to help projects and centers decide what tools to consider.

 

6. Lessons Learned

6.1 NASA Lessons Learned

No Lessons Learned have currently been identified for this requirement.

6.2 Other Lessons Learned

No other Lessons Learned have currently been identified for this requirement.

7. Software Assurance

SWE-189 - Code Coverage Measurements
4.5.9 The project manager shall ensure that the code coverage measurements for the software are selected, implemented, tracked, recorded, and reported.

7.1 Tasking for Software Assurance

  1. Confirm that code coverage measurements have been selected, performed, tracked, recorded, and communicated with each release.

7.2 Software Assurance Products

  • SA risk analysis and rationale on any uncovered software code percentage.


    Objective Evidence

    • Code coverage metric data
     Definition of objective evidence

    Objective evidence is an unbiased, documented fact showing that an activity was confirmed or performed by the software assurance/safety person(s). The evidence for confirmation of the activity can take any number of different forms, depending on the activity in the task. Examples are:

    • Observations, findings, issues, risks found by the SA/safety person and may be expressed in an audit or checklist record, email, memo or entry into a tracking system (e.g. Risk Log).
    • Meeting minutes with attendance lists or SA meeting notes or assessments of the activities and recorded in the project repository.
    • Status report, email or memo containing statements that confirmation has been performed with date (a checklist of confirmations could be used to record when each confirmation has been done!).
    • Signatures on SA reviewed or witnessed products or activities, or
    • Status report, email or memo containing Short summary of information gained by performing the activity. Some examples of using a “short summary” as objective evidence of a confirmation are:
      • To confirm that: “IV&V Program Execution exists”, the summary might be: IV&V Plan is in draft state. It is expected to be complete by (some date).
      • To confirm that: “Traceability between software requirements and hazards with SW contributions exists”, the summary might be x% of the hazards with software contributions are traced to the requirements.

7.3 Metrics

  • Software code/test coverage percentages for all identified safety-critical components (e.g., # of paths tested vs. total # of possible paths)
  • # of Source Lines of Code (SLOC) tested vs. total # of SLOC

7.4 Guidance

Step 1 - Confirm that engineering and the project have selected code coverage measurements, are performing, tracking, recording, and communicating the selected code coverage measurements with each release.

Step 2 - Identify uncovered software code by analyzing the code coverage measurement data.

Step 3 - Develop a risk assessment with engineering and rationale on any uncovered software code components.

Test coverage is a measurement that determines which statements in a body of code have been executed when a test suite is run, and which code has not been executed. This can help determine how well the code has been tested, whether enough testing has been done, and help maintain the code and test quality over the lifecycle. While code coverage tells how much of the code has been executed during testing, it does not mean code has been tested under every situation.

The recommended code coverage for Class A and B software is 100%. Classes C and D also are required to collect the code coverage measurements but can select a percentage of coverage that is agreed upon by the project and the Center’s Engineering Technical Authority. See the code coverage chart in the software guidance tab of this requirement.

Software assurance will confirm that the software projects are selecting a code coverage percentage goal, collecting the code coverage measurements, tracking them throughout the life cycle, recording the measurements, and reporting the code coverage measurements to the project, software assurance, and management. 

When the code coverage percentage is less than 100%, software assurance will analyze the code that has not been tested and identify the risks associated with leaving it untested. The analysis of the untested code should provide a rationale for why it is acceptable/unacceptable to leave that code untested. The typical reasons for code being untested are:

  • Requirement missing - the code that hasn’t been covered is performing an essential activity, but no requirement indicates that this should be done
  • Test missing - the code that hasn’t been covered relates to an existing requirement, but no test was implemented for it
  • Extraneous/dead code - The code that hasn’t been covered is not traceable to any requirement and isn’t needed by the software
  • Deactivated code - the code that hasn’t been covered isn’t traceable to any requirements for the current system but is intended to be executed in certain configurations.

The reasons why the code was not tested should be reported to the software manager and project manager along with the risk of the untested code. In the first two cases, the project may decide to add a requirement for a missing one or add a test or two to improve the testing (and the coverage!) It may be acceptable to leave the code in the two remaining cases untested, but the project manager may want to ensure that such extraneous/dead code or deactivated code does not get executed accidentally.



  • No labels

0 Comments