Page History

{alias:SWE-065}
{tabsetup:1. The Requirement|2. Rationale|3. Guidance|4. Small Projects|5. Resources|6. Lessons Learned}

{div3:id=tabs-1}

h1. 1. Requirements

3.4.1 The project shall establish and maintain:
        a.    Software Test Plan(s).
        b.    Software Test Procedure(s).
        c.    Software Test Report(s).

h2. {color:#003366}{*}1.1 Notes{*}{color}

The requirements for the content of a Software Test Plan, Software Test Procedure, and Software Test Report are defined in Chapter 5.

h2. 1.2 Applicability Across Classes

Class D and Not Safety Critical and class G are labeled with "P (Center)". P (Center) means that an approved Center-defined process which meets a non-empty subset of the full requirement can be used to achieve this requirement.

{applicable:asc=1\|ansc=1\|bsc=1\|bnsc=1\|csc=1\|cnsc=1\|dsc=1\|dnsc=p\|esc=1\|ensc=0\|f=1\|g=p|h=0|}
{div3}
{div3:id=tabs-2}

h1. 2. Rationale

Having plans and procedures in place increases the likelihood that all necessary and required tasks are performed and performed consistently. Development of plans and procedures provides the opportunity for stakeholders to give input and assist with the documentation and tailoring of the planned testing activities to ensure the outcome will meet the expectations and goals of the project. Test reports ensure that results of verification activities are documented and stored in the configuration management system for use in acceptance reviews or readiness reviews.

{panel}Ensuring the test plans, procedures and reports follow templates ensures consistency of documents across projects, ensures proper planning occurs, ensure proper activity and results are captured, and prevents repeating problems of the past. {panel}
{div3}
{div3:id=tabs-3}

h1. 3. Guidance

Projects should create test plans, procedures and reports following the content requirements in [SWE-104|SWE-104], [SWE-114|SWE-114], and [SWE-118|SWE-118].  This handbook provides guidance for each of these NPR 7150.2A requirements that address creating, reviewing, and obtaining approval for all of these test documents.

Once these documents are created, they should be maintained to reflect current project status, progress, and plans, which will change over the life of the project.  When requirements change ([SWE-071|SWE-071]), test plans, procedures and the resulting test reports may also need to be updated or revised to reflect the changes.  Changes to test plans and procedures may result from:
* Inspections / peer reviews of documentation
* Inspections / peer reviews of code
* Design changes
* Code maturation and changes (e.g., code changes to correct bugs or problems found during testing, interfaces revised during development)
* Availability of relevant test tools that were not originally part of the test plan (e.g., tools freed up from another project, funding becomes available to purchase new tools)
* Updated software hazards and mitigations (e.g., new hazards identified, hazards eliminated, mitigations are added or revised)
* Execution of the tests (e.g., issues found in test procedures)
* Test report / results analysis (e.g., incomplete, insufficient requirements coverage)
* Changes in test objectives or scope
* Changes to schedule, milestone, or budget changes
* Changes in test resource numbers or availability (e.g., personnel, tools, facilities)
* Changes to software classification or safety criticality (e.g., a research project not intended for flight becomes destined for use on the ISS)
* Process improvements relevant to test activities
* Changes in the project that affects the software testing effort

Just as the initial test plans, procedures, and reports require review and approval before use, the project team should ensure that updates are also reviewed and approved following project procedures.

Maintaining accurate and current test plans, procedures, and reports continues into the operation and maintenance phases of a project.

{note}Consult Center Process Asset Libraries (PALs) for Center-specific guidance and resources related to test plans, procedures and reports, including templates and examples. {note}


Additional guidance related to test plans, procedures, and reports may be found in the following related requirements in this handbook:
| *[SWE-071|SWE-071]* | Update Plans & Procedures |
| *[SWE-104|SWE-104]**:* | Software Test Plan |
| *[SWE-114|SWE-114]* | Software Test Procedures |
| *[SWE-118|SWE-118]**:* | Software Test Report |
\\
{div3}
{div3:id=tabs-4}

h1. 4. Small Projects

There is currently no guidance specific to small projects for this requirement.
{div3}
{div3:id=tabs-5}

h1. 5. Resources

# NASA Technical Standard, "[NASA Software Safety Guidebook|https://standards.nasa.gov/documents/detail/3315126]", NASA-GB-8719.13, 2004.
# IEEE Computer Society, "IEEE Standard for Software Verification and Validation", Chapter 7, IEEE STD 1012-2004, 2004. A user account is required to access IEEE standards via this [NASA Technical Standards System|http://standards.nasa.gov/] link.
# IEEE Computer Society, "IEEE Guide for Software Verification and Validation Plans", IEEE STD 1059-1993, 1993.  A user account is required to access IEEE standards via this [NASA Technical Standards System|http://standards.nasa.gov/] link.



h2. {toolstable}


{div3}
{div3:id=tabs-6}

h2. 6. Lessons Learned

The NASA Lessons Learned database contains the following lessons learned related to on insufficiencies in software test plans:
* *Procedures should be complete:* "The Aquarius Reflector test procedure lacked complete instructions for configuring the controller software prior to the test...The roles and responsibilities of the various personnel involved in the Aquarius acoustic test operations were not clearly documented.  This could lead to confusion during test operations." ([http://www.nasa.gov/offices/oce/llis/imported_content/lesson_2419.html|http://www.nasa.gov/offices/oce/llis/imported_content/lesson_2419.html])
* *Special measures needed for potentially hazardous tests*: "When planning tests that are potentially hazardous to personnel, flight hardware or facilities (e.g., high/low temperatures or pressure, stored energy, deployables), special measures should be taken to ensure that:  Test procedures are especially well written, well organized, and easy to understand by both engineering and quality assurance personnel.
*# Known test anomalies that history has shown to be inherent to the test equipment or conditions (including their likely causes, effects, and remedies) are documented and included in pre-test training. 
*# Readouts of safety-critical test control data are provided in an easily understood form (e.g., audible, visible or graphic format).
*# Test readiness reviews are held, and test procedures require confirmation that GSE test equipment and sensors have been properly maintained.
*# Quality assurance personnel are present and involved throughout the test to ensure procedures are properly followed, including prescribed responses to pre-identified potential anomalies."
([https://nen.nasa.gov/llis_content/0991.html|https://nen.nasa.gov/llis_content/0991.html])

* *Test plans should reflect proper configurations*: "Testing of the software changes was inadequate at the Unit, Integrated and Formal test level.  In reviewing test plans...neither had test steps where BAT06 and BAT04 were running concurrently in a launch configuration scenario.  Thus no test runs were done with the ... program that would reflect the new fully loaded console configuration.  Had the launch configuration scenarios been included in integrated and acceptance testing, this might have revealed the code timing problems." ([https://nen.nasa.gov/llis_lib/pdf/1035716main_PR%20LCA%204168.pdf|https://nen.nasa.gov/llis_lib/pdf/1035716main_PR%20LCA%204168.pdf])
* *Include test monitoring software safety steps*: "Prior to test, under the test principle of 'First, Do No Harm' to flight equipment, assure that test monitoring and control software is programmed or a limiting hardware device is inserted to prevent over-test under all conditions..." ([http://www.nasa.gov/offices/oce/llis/1529.html|http://www.nasa.gov/offices/oce/llis/1529.html])
\\
{div3}
{tabclose}