1. Requirements

4.5.10 The project manager shall verify code coverage is measured by analysis of the results of the execution of tests.

1.1 Notes

If it can be justified that the required percentage cannot be achieved by test execution, the analysis, inspection, or review of design can be applied to the non-covered code. The goal of the complementary analysis is to assess that the non-covered code behavior is as expected.

1.2 History

1.3 Applicability Across Classes

Key:    - Applicable | - Not Applicable

2. Rationale

To identify which lines of source code have been tested and which lines of your source code have not been tested and to provide data showing the completeness of the executed software tests.

3. Guidance

Test coverage is a measure used to describe the degree to which the source code of a program is executed when a particular test suite runs. A program with high test coverage, measured as a percentage, has had more of its source code executed during testing, which suggests it has a lower chance of containing undetected software bugs compared to a program with low test coverage. Many different metrics can be used to calculate test coverage; some of the most basic are the percentage of program subroutines and the percentage of program statements called during execution of the test suite.

Basic coverage criteria

There are several coverage criteria, the main ones being:

• Function coverage – Has each function (or subroutine) in the program been called?
• Statement coverage – Has each statement in the program been executed?
• Edge coverage – has every edge in the Control flow graph been executed?
• Branch coverage – Has each branch of each control structure (such as in if and case statements) been executed? For example, given an, if statement, have both the true and false branches been executed?
• Condition coverage (or predicate coverage) – Has each Boolean sub-expression evaluated both to true and false?

Code coverage measurements are defined, tracked, recorded, and reported in SWE-189.  The project manager should verify that the code coverage is measured by analysis of the results of the execution tests.  Code coverage analysis is the process of (1) finding areas of a program not exercised by a set of test cases, (2) creating additional test cases to increase coverage, and (3) determining a quantitative measure of code coverage, which is an indirect measure of quality.  

Code coverage is supported for specific languages by tools specific to that language.  Not all languages are supported.  Code coverage analysis during the unit and component testing can be implemented manually or by an analysis tool.  As the source code gets deeper into an embedded system, manual code coverage becomes problematic.  When the software is embedded in hardware, the tools’ ability to modify the source code, return the data through the hardware interfaces, and meet hardware performance requirements is an expensive effort.

Within the 5.08 - SDP-SMP - Software Development - Management Plan, define the code coverage metric to be used, and the metric goal percentage of the testing suite rigor.  Code coverage metrics and goals can differ for specific source code sections requiring more rigor.

Within the 5.10 - STP - Software Test Plan, include the code coverage metrics as part of the test results.  The code coverage metrics should include a reason for the source code lines reported as not covered in testing. 

Guidance for systems that are in operation and maintenance.

Code coverage is part of the development and testing process. For existing systems, all modifications should assess code coverage and any regression testing and requalification software testing should include code coverage metrics.

Additional guidance related to code coverage may be found in the following related requirements in this Handbook:

4. Small Projects

No additional guidance is available for small projects.

5. Resources

5.1 References

5.2 Tools

6. Lessons Learned

6.1 NASA Lessons Learned

No Lessons Learned have currently been identified for this requirement.

6.2 Other Lessons Learned

No other Lessons Learned have currently been identified for this requirement.

7. Software Assurance

7.1 Tasking for Software Assurance

1. Confirm that the project performs code coverage analysis using the results of the tests or by use of a code coverage tool.

2. Analyze the code coverage measurements for uncovered software code.

3. Assess any uncovered software code for potential risk, issues, or findings. 

7.2 Software Assurance Products

• Verification Activities Analysis
• Software assurance results of code coverage analysis.
• Potential risks, issues resulting from uncovered code during testing.

  
  

  Objective Evidence

  • Code coverage metric data
   Definition of objective evidence

  Objective evidence is an unbiased, documented fact showing that an activity was confirmed or performed by the software assurance/safety person(s). The evidence for confirmation of the activity can take any number of different forms, depending on the activity in the task. Examples are:

  • Observations, findings, issues, risks found by the SA/safety person and may be expressed in an audit or checklist record, email, memo or entry into a tracking system (e.g. Risk Log).
  • Meeting minutes with attendance lists or SA meeting notes or assessments of the activities and recorded in the project repository.
  • Status report, email or memo containing statements that confirmation has been performed with date (a checklist of confirmations could be used to record when each confirmation has been done!).
  • Signatures on SA reviewed or witnessed products or activities, or
  • Status report, email or memo containing a short summary of information gained by performing the activity. Some examples of using a “short summary” as objective evidence of a confirmation are:
    • To confirm that: “IV&V Program Execution exists”, the summary might be: IV&V Plan is in draft state. It is expected to be complete by (some date).
    • To confirm that: “Traceability between software requirements and hazards with SW contributions exists”, the summary might be x% of the hazards with software contributions are traced to the requirements.
  • The specific products listed in the Introduction of 8.16 are also objective evidence as well as the examples listed above.

7.3 Metrics

• # of Source Lines of Code (SLOC) tested vs. total # of SLOC
• # of tests successfully completed vs. total # of tests
• Software code/test coverage percentages for all identified safety-critical components (e.g., # of paths tested vs. total # of possible paths)
• # of Non-Conformances identified during each testing phase (Open, Closed, Severity)
• # of tests executed vs. # of tests successfully completed
• # of Risks trending up over time
• # of Risks trending down over time
• # of Risks with mitigation plans vs. total # of Risks
• # of Risks by Severity (e.g., red, yellow, green) over time
• # of Risks identified in each life-cycle phase (Open, Closed)

7.4 Guidance

Software assurance will confirm that the project is measuring its code coverage by analyzing the performance of the tests. One method of confirming this is to take some spot checks of what the project personnel is doing to get their code coverage measurements. (They may be taking the tests to run and using the trace matrix to see what code the test covers and then analyzing the test results to determine if that code was executed.)

Guidance for systems that are in operation and maintenance.

Code coverage is part of the development and testing process. For existing systems, all modifications should assess code coverage and any regression testing and requalification software testing should include code coverage metrics.