1. Requirements

1.1 Notes

This requirement can be met by running unit, integration, and validation tests; measuring the code coverage; and achieving the code coverage by additional (requirement based) tests, inspection, or analysis. If the project does not get 100 percent structural coverage, it means one of four things and each requires action on the project manager’s part: requirement missing

- the code that hasn’t been covered is performing an essential activity, but no requirement indicates that this should be done; test missing

- the code that hasn’t been covered relates to an existing requirement, but no test was implemented for it; extraneous/dead code

– the code that hasn’t been covered is not traceable to any requirement and isn’t needed by the software; deactivated code

- the code that hasn’t been covered isn’t traceable to any requirements for the current system, but is intended to be executed in specific configurations. The code coverage data and any rationale for uncovered code should be presented and reviewed at major project milestones.

1.2 History

1.3 Applicability Across Classes

2. Rationale

3. Guidance

Code coverage is an important aspect of good engineering practice. Code coverage is a software testing metric that shows you which lines of your source code have been tested and which lines of your source code have not been tested. This insight allows you to improve your testing so you can improve your code. Code coverage shows which areas of code have been tested ("covered") and which have not. Code coverage provides critical information to show teams where to focus their testing.

Code coverage measurement simply determines which statements in a body of code have been executed through a test run, and which statements have not.  Code coverage measurements for the software should be selected, implemented, tracked, recorded, and reported.

In summary, code coverage is measured for the following reasons:

• To know how well our tests test our code
• To know whether we have enough testing in place
• To maintain the test quality over the lifecycle of a project

If the project does not get 100% structural coverage, it means one of four things and each requires action on the project manager’s part:

• Requirement missing - the code that hasn’t been covered is performing an essential activity, but no requirement indicates that this should be done
• Test missing - the code that hasn’t been covered relates to an existing requirement, but no test was implemented for it
• Extraneous/dead code - The code that hasn’t been covered is not traceable to any requirement and isn’t needed by the software
• Deactivated code - the code that hasn’t been covered isn’t traceable to any requirements for the current system but is intended to be executed in certain configurations.

To measure what percentage of code has been exercised by a test suite, one or more coverage criteria are used. Coverage criteria are usually defined as rules or requirements, which a test suite needs to satisfy.

There are several coverage criteria, the main ones being:

• Function coverage – Has each function (or subroutine) in the program been called?
• Statement coverage – Has each statement in the program been executed?
• Branch coverage – Has each branch of each control structure (such as in if and case statements) been executed? For example, given an, if statement, have both the true and false branches been executed? Another way of saying this is, has every edge in the program been executed?
• Condition coverage (or predicate coverage) – Has each Boolean sub-expression evaluated both to true and false?

 Below is a suggestion on code coverage for software classes.

Additional guidance related to code coverage may be found in the following related requirements in this Handbook:

4. Small Projects

No additional guidance is available for small projects.

5. Resources

5.1 References

5.2 Tools

6. Lessons Learned

6.1 NASA Lessons Learned

No Lessons Learned have currently been identified for this requirement.

6.2 Other Lessons Learned

No other Lessons Learned have currently been identified for this requirement.

7. Software Assurance

7.1 Tasking for Software Assurance

1. Confirm that code coverage measurements have been selected, performed, tracked, recorded, and communicated with each release.

7.2 Software Assurance Products

• SA risk analysis and rationale on any uncovered software code percentage.

  
  

  Note
  title Objective Evidence
  Code coverage metric data

  Expand
  title Definition of objective evidence
  Include Page SITE:Definition of Objective Evidence SITE:Definition of Objective Evidence

7.3 Metrics

• Software code/test coverage percentages for all identified safety-critical components (e.g., # of paths tested vs. total # of possible paths)
• # of Source Lines of Code (SLOC) tested vs. total # of SLOC

7.4 Guidance

Step 1 - Confirm that engineering and the project have selected code coverage measurements, are performing, tracking, recording, and communicating the selected code coverage measurements with each release.

Step 2 - Identify uncovered software code by analyzing the code coverage measurement data.

Step 3 - Develop a risk assessment with engineering and rationale on any uncovered software code components.

Test coverage is a measurement that determines which statements in a body of code have been executed when a test suite is run, and which code has not been executed. This can help determine how well the code has been tested, whether enough testing has been done, and help maintain the code and test quality over the lifecycle. While code coverage tells how much of the code has been executed during testing, it does not mean code has been tested under every situation.

The recommended code coverage for Class A and B software is 100%. Classes C and D also are required to collect the code coverage measurements but can select a percentage of coverage that is agreed upon by the project and the Center’s Engineering Technical Authority. See the code coverage chart in the software guidance tab of this requirement.

Software assurance will confirm that the software projects are selecting a code coverage percentage goal, collecting the code coverage measurements, tracking them throughout the life cycle, recording the measurements, and reporting the code coverage measurements to the project, software assurance, and management. 

When the code coverage percentage is less than 100%, software assurance will analyze the code that has not been tested and identify the risks associated with leaving it untested. The analysis of the untested code should provide a rationale for why it is acceptable/unacceptable to leave that code untested. The typical reasons for code being untested are:

• Requirement missing - the code that hasn’t been covered is performing an essential activity, but no requirement indicates that this should be done
• Test missing - the code that hasn’t been covered relates to an existing requirement, but no test was implemented for it
• Extraneous/dead code - The code that hasn’t been covered is not traceable to any requirement and isn’t needed by the software
• Deactivated code - the code that hasn’t been covered isn’t traceable to any requirements for the current system but is intended to be executed in certain configurations.

The reasons why the code was not tested should be reported to the software manager and project manager along with the risk of the untested code. In the first two cases, the project may decide to add a requirement for a missing one or add a test or two to improve the testing (and the coverage!) It may be acceptable to leave the code in the two remaining cases untested, but the project manager may want to ensure that such extraneous/dead code or deactivated code does not get executed accidentally.