1. Requirements

1.1 Notes

If it can be justified that the required percentage cannot be achieved by test execution, the analysis, inspection, or review of design can be applied to the non-covered code. The goal of the complementary analysis is to assess that the non-covered code behavior is as expected.

This requirement can be met by running unit, integration, and validation tests; measuring the code coverage; and achieving the code coverage by additional (requirement based) tests, inspection, or analysis. (For more guidance in using test coverage to improve the code, see Tab 3 of SWE-190 in the Software Engineering Handbook (NASA-HDBK-2203)).

The code coverage data and any rationale for uncovered code should be presented and reviewed at major project milestones. 

1.2 History

1.3 Applicability Across Classes

2. Rationale

Code coverage helps to identify which lines of source code have been tested and which lines of your source code have not been tested and to provide data showing the completeness of the executed software tests. This helps to identify what areas of the code are needed and what areas may not be needed or are only used in specific scenarios. Code coverage can provide a profile of what software gets tested the most and the metrics can help guide where more involved or rigorous testing needs to occur, potentially at other levels of the system. 

3. Guidance

3.1 Test Coverage

Code coverage is an important aspect of good engineering practice. Code coverage is a software testing metric that shows you which lines of your source code have been tested (“covered”) and which lines of your source code have not been tested. This insight allows you to improve your testing so you can improve your code. Code coverage provides critical information to show teams where to focus their testing. 

Code coverage measurement simply determines which statements in a body of code have been executed through a test run, and which statements have not.  Code coverage measurements for the software should be selected, implemented, tracked, recorded, and reported. 

In summary, code coverage is measured for the following reasons: 

• To know how well our tests test our code 

• To know whether we have enough testing in place 

• To maintain the test quality over the life cycle of a project 

Test coverage is a measure used to describe the degree to which the source code of a program is executed when a particular test suite runs. A program with high test coverage, measured as a percentage, has had more of its source code executed during testing, which suggests it has a lower chance of containing undetected software bugs compared to a program with low test coverage. Many different metrics can be used to calculate test coverage; some of the most basic is the percentage of program subroutines and the percentage of program statements called during the execution of the test suite.

If the project does not get 100% structural coverage, it means one of four things and each requires action on the project manager’s part: 

• Requirement missing - the code that hasn’t been covered is performing an essential activity, but no requirement indicates that this should be done 

• Test missing - the code that hasn’t been covered relates to an existing requirement, but no test was implemented for it 

• Extraneous/dead code - The code that hasn’t been covered is not traceable to any requirement and isn’t needed by the software 

• Deactivated code - the code that hasn’t been covered isn’t traceable to any requirements for the current system but is intended to be executed in certain configurations. 

See also Topic 7.06 - Software Test Estimation and Testing Levels, 7.21 - Multi-condition Software Requirements, 8.19 - Dead / Dormant Code and Safety-Critical Software, 8.20 - Safety Specific Activities in Each Phase, SWE-062 - Unit Test, 

3.2 Basic Coverage Criteria

To measure what percentage of code has been exercised by a test suite, one or more coverage criteria are used. Coverage criteria are usually defined as rules or requirements, which a test suite needs to satisfy. 

There are several coverage criteria, the main ones being: 

• Function coverage – Has each function (or subroutine) in the program been called? 

• Statement coverage – Has each statement in the program been executed? 
• Branch coverage – Has each branch of each control structure (such as in if and case statements) been executed? For example, given an, if statement, have both the true and false branches been executed? Another way of saying this is, has every edge in the program been executed? 
• Edge coverage – has every edge in the Control flow graph been executed?
• Condition coverage (or predicate coverage) – Has each Boolean sub-expression evaluated both to true and false? 

 Below is a suggestion on code coverage for software classes. 

Also, note that coverage of libraries/objects vary based on the project’s classification and safety criticality. For example, if a math library is utilized some functions may not be utilized and could go untested. 

3.3 Code Coverage Measurements

Code coverage measurements are defined, tracked, recorded, and reported in SWE-189 - Code Coverage Measurements.  The project manager should verify that the code coverage is measured by analysis of the results of the execution tests.  Code coverage analysis is the process of (1) finding areas of a program not exercised by a set of test cases, (2) creating additional test cases to increase coverage, and (3) determining a quantitative measure of code coverage, which is an indirect measure of quality.  

3.4 Code Coverage Tools

Code coverage is supported for specific languages by tools specific to that language.  Not all languages are supported.  Code coverage analysis during the unit and component testing can be implemented manually or by an analysis tool.  As the source code gets deeper into an embedded system, manual code coverage becomes problematic.  When the software is embedded in hardware, the tools’ ability to modify the source code, return the data through the hardware interfaces, and meet hardware performance requirements is an expensive effort.

Within the 5.08 - SDP-SMP - Software Development - Management Plan, define the code coverage metric to be used, and the metric goal percentage of the testing suite rigor.  Code coverage metrics and goals can differ for specific source code sections requiring more rigor.

Within the 5.10 - STP - Software Test Plan, include the code coverage metrics as part of the test results.  The code coverage metrics should include a reason for the source code lines reported as not covered in testing. 

See also SWE-135 - Static Analysis, SWE-219 - Code Coverage for Safety Critical Software, 

3.5 Guidance For Systems That Are In Operation And Maintenance

Code coverage is part of the development and testing process. For existing systems, all modifications should assess code coverage and any regression testing and requalification software testing should include code coverage metrics.

3.6 Additional Guidance

Additional guidance related to this requirement may be found in the following materials in this Handbook:

3.7 Center Process Asset Libraries

See the following link(s) in SPAN for process assets from contributing Centers (NASA Only). 

4. Small Projects

For small projects, the rigor may be less. The core of the code should have significant coverage (including key decision points). The code coverage percentage should be commensurate with the risk posture of the project. Use of a static analysis tool that works with, or is embedded in, a continuous integration tool can save time and provide all the information you need on code coverage within your test suite. Any untested sections of code should have some rationale documented to ensure that the project is aware of this and agrees with the approach. 

5. Resources

5.1 References

Click here to view master references table. 

No references have been currently identified for this SWE. If you wish to suggest a reference, please leave a comment below. 

5.2 Tools 

Tools to aid in compliance with this SWE, if any, may be found in the Tools Library in the NASA Engineering Network (NEN).  

NASA users find this in the Tools Library in the Software Processes Across NASA (SPAN) site of the Software Engineering Community in NEN.  

The list is informational only and does not represent an “approved tool list”, nor does it represent an endorsement of any particular tool.  The purpose is to provide examples of tools being used across the Agency and to help projects and centers decide what tools to consider. 

5.2 Tools

6. Lessons Learned

6.1 NASA Lessons Learned

No Lessons Learned have currently been identified for this requirement.

6.2 Other Lessons Learned

No other Lessons Learned have currently been identified for this requirement.

7. Software Assurance

7.1 Tasking for Software Assurance

7.2 Software Assurance Products

• SA risk analysis and rationale on any uncovered software code percentage. 
• Verification Activities Analysis
• Software assurance results of code coverage analysis.
• Potential risks, and issues resulting from uncovered code during testing.

  
  

  Code coverage metric data

7.3 Metrics

• # of Source Lines of Code (SLOC) tested vs. total # of SLOC
• # of tests completed vs. total # of tests
• Software code/test coverage percentages for all identified safety-critical components (e.g., # of paths tested vs. total # of possible paths)
• # of Non-Conformances identified during each testing phase (Open, Closed, Severity)
• # of tests executed vs. # of tests completed
• # of Risks trending up over time
• # of Risks trending down over time
• # of Risks with mitigation plans vs. total # of Risks
• # of Risks by Severity (e.g., red, yellow, green) over time
• # of Risks identified in each life cycle phase (Open, Closed)

 Note: Metrics in bold type are required by all projects

See also Topic 8.18 - SA Suggested Metrics.

7.4 Guidance

Software assurance will confirm that the project is measuring its code coverage by analyzing the performance of the tests. One method of confirming this is to take some spot checks of what the project personnel is doing to get their code coverage measurements. (They may be taking the tests to run and using the trace matrix to see what code the test covers and then analyzing the test results to determine if that code was executed.)

7.4.1 Guidance For Systems That Are In Operation And Maintenance

Code coverage is part of the development and testing process. For existing systems, all modifications should assess code coverage and any regression testing and requalification software testing should include code coverage metrics.

Step 1 - Confirm that engineering and the project have selected code coverage measurements, are performing, tracking, recording, and communicating the selected code coverage measurements with each release. 

Step 2 - Identify uncovered software code by analyzing the code coverage measurement data. 

Step 3 - Develop a risk assessment with engineering and rationale on any uncovered software code components. 

Test coverage is a measurement that determines which statements in a body of code have been executed when a test suite is run, and which code has not been executed. This can help determine how well the code has been tested, whether enough testing has been done, and help maintain the code and test quality over the life cycle. While code coverage tells how much of the code has been executed during testing, it does not mean code has been tested under every situation. 

The recommended code coverage for Class A and B software is 100%. Classes C and D also are required to collect the code coverage measurements but can select a percentage of coverage that is agreed upon by the project and the Center’s Engineering Technical Authority. See the code coverage chart in the software guidance tab of this requirement. 

Software assurance will confirm that the software projects are selecting a code coverage percentage goal, collecting the code coverage measurements, tracking them throughout the life cycle, recording the measurements, and reporting the code coverage measurements to the project, software assurance, and management.  

When the code coverage percentage is less than 100%, software assurance will analyze the code that has not been tested and identify the risks associated with leaving it untested. The analysis of the untested code should provide a rationale for why it is acceptable/unacceptable to leave that code untested. The typical reasons for code being untested are: 

• Requirement missing - the code that hasn’t been covered is performing an essential activity, but no requirement indicates that this should be done. 

• Test missing - the code that hasn’t been covered relates to an existing requirement, but no test was implemented for it. 

• Extraneous/dead code - The code that hasn’t been covered is not traceable to any requirement and isn’t needed by the software. 

• Deactivated code - the code that hasn’t been covered isn’t traceable to any requirements for the current system but is intended to be executed in certain configurations. 

The reasons why the code was not tested should be reported to the software manager and project manager along with the risk of the untested code. In the first two cases, the project may decide to add a requirement for a missing one or add a test or two to improve the testing (and the coverage!) It may be acceptable to leave the code in the two remaining cases untested, but the project manager may want to ensure that such extraneous/dead code or deactivated code does not get executed accidentally. 

7.5 Additional Guidance

Additional guidance related to this requirement may be found in the following materials in this Handbook: