5.1.3.1 The Software Test Plan shall include: [SWE-104] a. Test levels (separate test effort that has its own documentation and resources, e.g., component, integration, and system testing). NPR 7150.2, NASA Software Engineering Requirements, does not include any notes for this requirement. Classes C-E, Safety Critical, are labeled with "P (Center) + SO." "P (Center)" means that an approved Center-defined process that meets a non-empty subset of the full requirement can be used to achieve this requirement. "SO" means that the requirement applies only for safety critical portions of the software. Class A_SC A_NSC B_SC B_NSC C_SC C_NSC D_SC D_NSC E_SC E_NSC F G H Applicable? X P(C) X P(C) X P(C) Key: A_SC = Class A Software, Safety-Critical | A_NSC = Class A Software, Not Safety-Critical | ... | - Applicable | - Not Applicable Planning the software testing activity allows thorough deliberation of tasks, methods, environments, and related criteria before they are implemented. Planning also allows the project team to improve a current project, based on lessons learned from previous projects, including using more appropriate or efficient techniques and ensuring the performance of steps previously missed or not included. As with any task, having a plan in place ensures that all necessary and required tasks are performed. Development of that plan provides the opportunity for stakeholders to give input and assist with the documentation and tailoring of the planned testing activities to ensure the outcome will meet the expectations and goals of the project. Ensuring the Software Test Plan follows a template and includes specified information ensures consistency of test plans across projects, ensures proper planning occurs, and prevents repeating problems of the past. The Software Test Plan may be standalone or part of the Software Management Plan. Follow Center policies and procedures when determining which approach to use for a particular project. The Software Test Plan may be tailored by software classification. Goddard Space Flight Center's (GSFC's) 580-STD-077-01, Requirements for Minimum Contents of Software Documents, 090 provides one suggestion for tailoring a Software Test Plan based on the required contents and the classification of the software being tested. As shown in the SWE-104 requirement text above, the Software Test Plan is required to contain specific information. Below are suggestions for the type of information to consider for the named required contents. Note that if this information is added to the Software Test Plan in phases, IEEE Std 1059-1993, IEEE Guide for Software Verification and Validation Plans, 211 notes that: "The requirements phase produces the ...acceptance and system test plans. The design phase produces the ... component and integration test plans." Test levels Specific types of testing may have their own plans, resources, tools, schedules, and other information particular to the testing being performed. Where plans exist to document this information, they need not be duplicated in the Software Test Plan, but they need to be identified and referenced to provide an overall view of the project's testing efforts. This section of the plan also describes the levels at which testing will be performed: configuration item level, system level, etc. Unit testing The intent of unit testing is to confirm that a unit performs the capability assigned to it, correctly interfaces with other units and data, and represents a faithful implementation of the unit design. 452 In accordance with IEEE Std 610.12-1990, IEEE Standard Glossary of Software Engineering Terminology, 222, a unit is defined as: (1) A separately testable element specified in the design of a computer software component. The Software Test Plan includes an overview of the unit testing process to be used for this project. The section(s) of the Software Test Plan focused on unit testing addresses the following: Software integration testing According to NASA-GB-8719.13, 276 "Integration testing deals with how the software components will be incorporated into the final software system and what will be tested at each integration step." The Software Test Plan includes an overview of the software integration testing process to be used for this project. The section(s) of the Software Test Plan focused on software integration testing addresses the following: Systems integration testing The Software Test Plan includes an overview of the testing process used to test software integrated into the larger system. The section(s) of the software test plan focused on systems integration testing addresses the following: End-to-end testing The Software Test Plan includes an overview of the process used for end-to-end testing. The section(s) of the Software Test Plan focused on end-to-end testing addresses the following: Acceptance testing Acceptance testing needs to be conducted after the appropriate readiness review has been successfully completed. This type of testing is the customer acceptance test, and the Software Test Plan includes an overview of the process used. The section(s) of the Software Test Plan focused on acceptance testing includes the following: Regression testing Regression tests are used to ensure that changes made to software have had no unintended side effects. The section(s) of the Software Test Plan focused on regression testing addresses the following: Test classes (designated grouping of test cases) Test classes describe the grouping of tests that will be performed throughout the various types of testing. When developing the Software Test Plan, consider: General test conditions General test conditions are conditions that apply to all of the planned tests or to a specific group of tests. When documenting general test conditions, consider statements and conditions, such as these taken from Langley Research Center's NPR 7150.2 Class A Required Testing Documents With Embedded Guidance: 083 Test progression Test progression addresses the sequence or ordering of tests. The Software Test Plan describes dependencies among tests that require that tests be performed in a particular order. Data recording, reduction, and analysis This section of the Software Test Pan describes processes and procedures for capturing and evaluating/analyzing results and issues found during all types of testing. Consider "manual, automatic, and semi-automatic techniques for recording test results, manipulating the raw results into a form suitable for evaluation, and retaining the results of data reduction and analysis." 401 For specific suggestions for carrying out these activities, see SWE-068 and SWE-069 in this Handbook. NASA-GB-8719.13 276 recommends test reports be created for unit tests of safety-critical items. Test coverage (breadth and depth) or other methods for ensuring sufficiency of testing If not addressed elsewhere in the Software Test Plan, a description of the methods to be used for ensuring sufficient test coverage are provided. Methods could include: Planned tests, including items and their identifiers If not already included in sections of the plan focused on specific types of testing (unit, integration, etc.), all planned tests, test cases, data sets, etc., that will be used for the project need to be identified in the Software Test Plan, along with the software items they will be used to test. Each item needs to have its own unique identifier to ensure proper execution and tracking of the planned tests. Consider the following information as information to capture for each test: Test schedules NASA-GB-8719.13 276 notes that: "Scheduling testing phases is always an art, and depends on the expected quality of the software product... Previous history, either of the development team or similar projects, can help determine how long testing will take. Some methods (such as error seeding and Halstead's defect metric) exist for estimating defect density (number of defects per unit of code) when historical information is not available." Information to consider for test schedules includes: Requirements traceability (or verification matrix) One of the key purposes of testing is to confirm that the requirements for the project have been met. The Software Test Plan includes a traceability matrix (see SWE-072) that maps tests to software requirements. If the matrix already exists, the test plan includes a reference to the matrix. Qualification testing environment, site, personnel, and participating organizations This section or topic in the Software Test Plan lists items such as these taken from NPR 7150.2A Class A Required Testing Documents With Embedded Guidance: 083 In addition to the information required above, test plans address the following information for all types of testing: If not identified elsewhere, the Software Test Plan identifies the metrics to be collected for each type of testing. Suggested metrics include: Consider the following best practices for Software Test Plans: Use the right Sources of Information (first column in table below) as appropriate for the project and for each type of testing, such as: *Unit Test *SW Integration Test *Systems Integration Test *End-to-End Test *Acceptance Test *Regression Test Software Requirements Specification (SRS) X X X X X Software Design Description (SDD) X X Design traceability X X Interface documents X X X X X X Draft user documentation X Code coverage analyzer specifications X Criticality analysis X Draft operating documents X X X Draft maintenance documents X Final operating documents X Final user documentation X Concept documents X X Requirements traceability X X X X Expected customer usage patterns and conditions X X X X *May be a separate test plan referenced in the overall Software Test Plan or part of the overall Software Test Plan. Consult Center Process Asset Libraries (PALs) for Center-specific guidance and resources, such as templates, related to Software Test Plans and their contents. Additional guidance related to software testing may be found in the following related requirements in this Handbook: Cost Estimation Test Plans, Procedures, Reports Perform Testing Verify Implementation Evaluate Test Results Document Defects and Track Software Test Procedures Software Test Report Software Test Plans are necessary for all software projects, but for projects with small budgets or small teams starting with an existing test plan from a project of a similar type and size could help reduce the time and effort required to produce a test plan for a new project. Working with someone experienced in writing test plans, perhaps from another project and on a short-term basis, could help the project team prepare the document in a timely fashion without overburdening team resources. Where applicable, the test plan could reference other project documents rather than reproduce their contents, avoiding duplication of effort and reducing maintenance activities. Since the Software Test Plan may be standalone or part of the Software Management Plan, incorporating the test plan into a larger project document may be useful for document tracking, review, etc. Follow Center policies and procedures when determining which approach to use for a particular project. The Software Test Plan may be tailored by software classification. 580-STD-077-01 090 provides one suggestion for tailoring a Software Test Plan based on the required contents and the classification of the software being tested. This tailoring could reduce the size of the Software Test Plan and, therefore, the time and effort to produce and maintain it. Tools to aid in compliance with this SWE, if any, may be found in the Tools Library in the NASA Engineering Network (NEN). NASA users find this in the Tools Library in the Software Processes Across NASA (SPAN) site of the Software Engineering Community in NEN. The list is informational only and does not represent an “approved tool list”, nor does it represent an endorsement of any particular tool. The purpose is to provide examples of tools being used across the Agency and to help projects and centers decide what tools to consider. The NASA Lessons Learned database contains the following lessons learned related to software test planning:
See edit history of this section
Post feedback on this section
1. Requirements
b. Test types:
(1) Unit testing.
(2) Software integration testing.
(3) Systems integration testing.
(4) End-to-end testing.
(5) Acceptance testing.
(6) Regression testing.
c. Test classes (designated grouping of test cases).
d. General test conditions.
e. Test progression.
f. Data recording, reduction, and analysis.
g. Test coverage (breadth and depth) or other methods for ensuring sufficiency of testing.
h. Planned tests, including items and their identifiers.
i. Test schedules.
j. Requirements traceability (or verification matrix).
k. Qualification testing environment, site, personnel, and participating organizations.1.1 Notes
1.2 Applicability Across Classes
X - Applicable with details, read above for more | P(C) - P(Center), follow center requirements or procedures2. Rationale
3. Guidance
(2) A logically separable part of a computer program.
(3) A software component that is not subdivided into other components.4. Small Projects
5. Resources
5.1 Tools
6. Lessons Learned
SWE-104 - Software Test Plan
Web Resources
View this section on the websiteUnknown macro: {page-info}
NPR 7150.2, section 5.1.3, states: "The Software Test Plan describes the plans for software component level testing, software integration testing, software qualification testing, and system qualification testing of software systems. The plan describes the software test environment, development, and test activities. The plan provides an overview of software testing, test schedules, and test management procedures."
0 Comments