bannerc

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Fixed SWE table in tab 3
Tabsetup
01. The Requirement
12. Rationale
23. Guidance
34. Small Projects
45. Resources
56. Lessons Learned
67. Software Assurance
Div
idtabs-1

1. Requirements

Excerpt

4.5.9 The project manager shall ensure that the code coverage measurements for the software are selected, implemented, tracked, recorded, and reported.

1.1 Notes

This requirement can be met by running unit, integration, and validation tests; measuring the code coverage; and achieving the code coverage by additional (requirement based) tests, inspection, or analysis. If the project does not get 100 percent structural coverage, it means one of four things and each requires action on the project manager’s part: requirement missing

- the code that hasn’t been covered is performing an essential activity, but no requirement indicates that this should be done; test missing

- the code that hasn’t been covered relates to an existing requirement, but no test was implemented for it; extraneous/dead code

– the code that hasn’t been covered is not traceable to any requirement and isn’t needed by the software; deactivated code

- the code that hasn’t been covered isn’t traceable to any requirements for the current system, but is intended to be executed in specific configurations. The code coverage data and any rationale for uncovered code should be presented and reviewed at major project milestones.

1.2 History

Click here to view the history of this requirement: SWE-189 History

1.3 Applicability Across Classes

 

Applicable c
a1
b1
csc1
c1
d1
dsc1
e0
f0
g0
h0

Div
idtabs-2

2. Rationale

To identify which lines of source code have been tested and which lines of your source code have not been tested and to provide data showing the completeness of the executed software tests.
Div
idtabs-3

3. Guidance

Code coverage is an important aspect of good engineering practice. Code coverage is a software testing metric that shows you which lines of your source code have been tested and which lines of your source code have not been tested. This insight allows you to improve your testing so you can improve your code. Code coverage shows which areas of code have been tested ("covered") and which have not. Code coverage provides critical information to show teams where to focus their testing.

Code coverage measurement simply determines which statements in a body of code have been executed through a test run, and which statements have not.  Code coverage measurements for the software should be selected, implemented, tracked, recorded and reported.

In summary, code coverage is measured for the following reasons:

  • To know how well our tests actually test our code
  • To know whether we have enough testing in place
  • To maintain the test quality over the lifecycle of a project

If the project does not get 100% structural coverage, it means one of four things and each requires action on the project manager’s part:

  • Requirement missing - the code that hasn’t been covered is performing an essential activity, but no requirement indicates that this should be done
  • Test missing - the code that hasn’t been covered relates to an existing requirement, but no test was implemented for it
  • Extraneous/dead code - The code that hasn’t been covered is not traceable to any requirement and isn’t needed by the software
  • Deactivated code - the code that hasn’t been covered isn’t traceable to any requirements for the current system but is intended to be executed in certain configurations.

To measure what percentage of code has been exercised by a test suite, one or more coverage criteria are used. Coverage criteria are usually defined as rules or requirements, which a test suite needs to satisfy.

There are a number of coverage criteria, the main ones being:

  • Function coverage – Has each function (or subroutine) in the program been called?
  • Statement coverage – Has each statement in the program been executed?
  • Branch coverage – Has each branch of each control structure (such as in if and case statements) been executed? For example, given an, if statement, have both the true and false branches been executed? Another way of saying this is, has every edge in the program been executed?
  • Condition coverage (or predicate coverage) – Has each Boolean sub-expression evaluated both to true and false?

 Below is a suggestion on code coverage for software classes.

Code coverage versus Software Classification

A

B

C

D

Source code statement coverage

100%

100%

AM

AM

Source code decision coverage

100%

100%

AM

AM

Source code modified condition and decision coverage

100%

AM

AM

AM

NOTE: “AM“ means that the value is agreed and signed off by the Center’s Engineering TA and measured.

Additional guidance related to code coverage may be found in the following related requirements in this Handbook:

Div
idtabs-4

4. Small Projects

No additional guidance is available for small projects.

Div
idtabs-5

5. Resources

5.1 References

refstable
Show If
groupconfluence-users
Panel
titleColorred
titleVisible to editors only

Enter the necessary modifications to be made in the table below:

SWEREFs to be addedSWEREFS to be deleted


SWEREFs called out in text: none


5.2 Tools

Include Page
Tools Table Statement
Tools Table Statement
 

Div
idtabs-6

6. Lessons Learned

6.1 NASA Lessons Learned

No Lessons Learned have currently been identified for this requirement.

6.2 Other Lessons Learned

No other Lessons Learned have currently been identified for this requirement.

Div
idtabs-7

7. Software Assurance

Excerpt Include
SWE-189 - Code Coverage Measurements
SWE-189 - Code Coverage Measurements

7.1 Tasking for Software Assurance

  1. Confirm that code coverage measurements have been selected, performed, tracked, recorded, and communicated with each release.

7.2 Software Assurance Products

  • SA risk analysis and rationale on any uncovered software code percentage.


    Note
    titleObjective Evidence

    Evidence confirming that code coverage measurements have been selected, performed, tracked, recorded, and communicated with each release.

    Expand
    titleDefinition of objective evidence

    Include Page
    Definition of Objective Evidence
    Definition of Objective Evidence

7.3 Metrics

  • Software code/test coverage percentages for all identified safety-critical components (e.g., # of paths tested vs. total # of possible paths)
  •  # of Source Lines of Code (SLOC) tested vs. total # of SLOC

7.4 Guidance

Step 1 - Confirm that engineering and the project have selected code coverage measurements, are performing, tracking, recording, and communicating the selected code coverage measurements with each release.

Step 2 - Identify uncovered software code by analyzing the code coverage measurement data.

Step 3 - Develop a risk assessment with engineering and rationale on any uncovered software code components.

Test coverage is a measurement that determines which statements in a body of code have been executed when a test suite is run, and which code has not been executed. This can help determine how well the code has been tested, whether enough testing has been done and help maintain the code and test quality over the lifecycle. While code coverage tells how much of the code has been executed during testing, it does not mean code has been tested under every situation.

The recommended code coverage for Class A and B software is 100%. Classes C and D also are required to collect the code coverage measurements but can select a percentage of coverage that is agreed upon by the project and the Center’s Engineering Technical Authority. See the code coverage chart in the software guidance tab of this requirement.

Software assurance will confirm that the software projects are selecting a code coverage percentage goal, collecting the code coverage measurements, tracking them throughout the life cycle, recording the measurements and reporting the code coverage measurements to the project, software assurance, and management. 

When the code coverage percentage is less than 100%, software assurance will analyze the code that has not been tested and identify the risks associated with leaving it untested. The analysis of the untested code should provide a rationale for why it is acceptable/unacceptable to leave that code untested. The typical reasons for code being untested are:

  • Requirement missing - the code that hasn’t been covered is performing an essential activity, but no requirement indicates that this should be done
  • Test missing - the code that hasn’t been covered relates to an existing requirement, but no test was implemented for it
  • Extraneous/dead code - The code that hasn’t been covered is not traceable to any requirement and isn’t needed by the software
  • Deactivated code - the code that hasn’t been covered isn’t traceable to any requirements for the current system but is intended to be executed in certain configurations.

The reasons why the code was not tested should be reported to the software manager and project manager along with the risk of the untested code. In the first two cases, the project may decide to add a requirement for a missing one or add a test or two to improve the testing (and the coverage!) It may be acceptable to leave the code in the two remaining cases untested, but the project manager may want to ensure that such extraneous/dead code or deactivated code does not get executed accidentally.

...