1. Requirements

4.5.9 The project manager shall ensure that the code coverage measurements for the software are selected, implemented, tracked, recorded, and reported.

1.1 Notes

This requirement can be met by running unit, integration, and validation tests; measuring the code coverage; and achieving the code coverage by additional (requirement based) tests, inspection, or analysis.

If the project does not get 100 percent structural coverage, it means one of four things and each requires action on the project manager’s part:

• Requirement missing - the code that hasn’t been covered is performing an essential activity, but no requirement indicates that this should be done;
• Test missing - the code that hasn’t been covered relates to an existing requirement, but no test was implemented for it;
• Extraneous/dead code – the code that hasn’t been covered is not traceable to any requirement and isn’t needed by the software;
• Deactivated code - the code that hasn’t been covered isn’t traceable to any requirements for the current system, but is intended to be executed in specific configurations.

The code coverage data and any rationale for uncovered code should be presented and reviewed at major project milestones.

1.2 History

1.3 Applicability Across Classes

Key:    - Applicable | - Not Applicable

2. Rationale

To identify which lines of source code have been tested and which lines of your source code have not been tested and to provide data showing the completeness of the executed software tests. This helps to identify what areas of the code are needed and what areas may not be needed or are only used in specific scenarios. Code coverage can provide a profile of what software gets tested the most and the metrics can help guide where more involved or rigorous testing needs to occur, potentially at other levels of the system.

3. Guidance

3.1 Code Coverage

Code coverage is an important aspect of good engineering practice. Code coverage is a software testing metric that shows you which lines of your source code have been tested and which lines of your source code have not been tested ("covered"). This insight allows you to improve your testing so you can improve your code.  Code coverage provides critical information to show teams where to focus their testing.

Code coverage measurement simply determines which statements in a body of code have been executed through a test run, and which statements have not.  Code coverage measurements for the software should be selected, implemented, tracked, recorded, and reported. (See SWE-190 - Verify Code Coverage. )

In summary, code coverage is measured for the following reasons:

• To know how well our tests test our code
• To know whether we have enough testing in place
• To maintain the test quality over the life cycle of a project

See also SWE-065 - Test Plan, Procedures, Reports, SWE-066 - Perform Testing, 

3.2 Dealing With Less Than 100% Coverage

If the project does not get 100% structural coverage, it means one of four things and each requires action on the project manager’s part:

• Requirement missing - the code that hasn’t been covered is performing an essential activity, but no requirement indicates that this should be done
• Test missing - the code that hasn’t been covered relates to an existing requirement, but no test was implemented for it
• Extraneous/dead code - The code that hasn’t been covered is not traceable to any requirement and isn’t needed by the software
• Deactivated code - the code that hasn’t been covered isn’t traceable to any requirements for the current system but is intended to be executed in certain configurations.

See also Topic 8.19 - Dead / Dormant Code and Safety-Critical Software, 

3.3 Coverage Criteria

To measure what percentage of code has been exercised by a test suite, one or more coverage criteria are used. Coverage criteria are usually defined as rules or requirements, which a test suite needs to satisfy.

There are several coverage criteria, the main ones being:

• Function coverage – Has each function (or subroutine) in the program been called?
• Statement coverage – Has each statement in the program been executed?
• Branch coverage – Has each branch of each control structure (such as in if and case statements) been executed? For example, given an, if statement, have both the true and false branches been executed? Another way of saying this is, has every edge in the program been executed?
• Condition coverage (or predicate coverage) – Has each Boolean sub-expression evaluated both to true and false?

 Below is a suggestion on code coverage for software classes.

Also, note that coverage of libraries/objects vary based on the project’s classification and safety criticality. For example, if a math library is utilized some functions may not be utilized and could go untested. 

3.4 Additional Guidance

Additional guidance related to this requirement may be found in the following materials in this Handbook:

3.5 Center Process Asset Libraries

See the following link(s) in SPAN for process assets from contributing Centers (NASA Only). 

4. Small Projects

5. Resources

5.1 References

5.2 Tools

6. Lessons Learned

6.1 NASA Lessons Learned

No Lessons Learned have currently been identified for this requirement.

6.2 Other Lessons Learned

No other Lessons Learned have currently been identified for this requirement.

7. Software Assurance

7.1 Tasking for Software Assurance

7.2 Software Assurance Products

• SA risk analysis and rationale on any uncovered software code percentage.

  
  

  Objective Evidence

  • Code coverage metric data
   Definition of objective evidence

  Objective evidence is an unbiased, documented fact showing that an activity was confirmed or performed by the software assurance/safety person(s). The evidence for confirmation of the activity can take any number of different forms, depending on the activity in the task. Examples are:

  • Observations, findings, issues, risks found by the SA/safety person and may be expressed in an audit or checklist record, email, memo or entry into a tracking system (e.g. Risk Log).
  • Meeting minutes with attendance lists or SA meeting notes or assessments of the activities and recorded in the project repository.
  • Status report, email or memo containing statements that confirmation has been performed with date (a checklist of confirmations could be used to record when each confirmation has been done!).
  • Signatures on SA reviewed or witnessed products or activities, or
  • Status report, email or memo containing a short summary of information gained by performing the activity. Some examples of using a “short summary” as objective evidence of a confirmation are:
    • To confirm that: “IV&V Program Execution exists”, the summary might be: IV&V Plan is in draft state. It is expected to be complete by (some date).
    • To confirm that: “Traceability between software requirements and hazards with SW contributions exists”, the summary might be x% of the hazards with software contributions are traced to the requirements.
  • The specific products listed in the Introduction of 8.16 are also objective evidence as well as the examples listed above.

7.3 Metrics

• Software code/test coverage percentages for all identified safety-critical components (e.g., # of paths tested vs. total # of possible paths)
• # of Source Lines of Code (SLOC) tested vs. total # of SLOC

 Note: Metrics in bold type are required by all projects

See also Topic 8.18 - SA Suggested Metrics.

7.4 Guidance

Step 1 - Confirm that engineering and the project have selected code coverage measurements, are performing, tracking, recording, and communicating the selected code coverage measurements with each release.

Step 2 - Identify uncovered software code by analyzing the code coverage measurement data.

Step 3 - Develop a risk assessment with engineering and rationale on any uncovered software code components.

Test coverage is a measurement that determines which statements in a body of code have been executed when a test suite is run, and which code has not been executed. This can help determine how well the code has been tested, whether enough testing has been done, and help maintain the code and test quality over the life cycle. While code coverage tells how much of the code has been executed during testing, it does not mean code has been tested under every situation.

The recommended code coverage for Class A and B software is 100%. Classes C and D also are required to collect the code coverage measurements but can select a percentage of coverage that is agreed upon by the project and the Center’s Engineering Technical Authority. See the code coverage chart in the software guidance tab of this requirement.

Software assurance will confirm that the software projects are selecting a code coverage percentage goal, collecting the code coverage measurements, tracking them throughout the life cycle, recording the measurements, and reporting the code coverage measurements to the project, software assurance, and management. 

When the code coverage percentage is less than 100%, software assurance will analyze the code that has not been tested and identify the risks associated with leaving it untested. The analysis of the untested code should provide a rationale for why it is acceptable/unacceptable to leave that code untested. The typical reasons for code being untested are:

• Requirement missing - the code that hasn’t been covered is performing an essential activity, but no requirement indicates that this should be done
• Test missing - the code that hasn’t been covered relates to an existing requirement, but no test was implemented for it
• Extraneous/dead code - The code that hasn’t been covered is not traceable to any requirement and isn’t needed by the software
• Deactivated code - the code that hasn’t been covered isn’t traceable to any requirements for the current system but is intended to be executed in certain configurations.

The reasons why the code was not tested should be reported to the software manager and project manager along with the risk of the untested code. In the first two cases, the project may decide to add a requirement for a missing one or add a test or two to improve the testing (and the coverage!) It may be acceptable to leave the code in the two remaining cases untested, but the project manager may want to ensure that such extraneous/dead code or deactivated code does not get executed accidentally.

7.5 Additional Guidance

Additional guidance related to this requirement may be found in the following materials in this Handbook: