3.4.2 The project shall perform software testing as defined in the Software Test Plan. A best practice for Class A, B, and C software projects is to have formal software testing conducted, witnessed, and approved by an independent organization outside of the development team. Testing could include software integration testing, systems integration testing, validation testing, end-to-end testing, acceptance testing, white and black box testing, decision and path analysis, statistical testing, stress testing, performance testing, regression testing, qualification testing, simulation, and others. Use of automated software testing tools are also to be considered in software testing. Test breadth and accuracy can be increased through the use of test personnel independent of the software design and implementation teams, software peer reviews/inspections of Software Test Procedures and Software Test Results, and employing impartial test witnesses. Class E Not Safety Critical and Class G are labeled with "P (Center)." This means that an approved Center-defined process which meets a non-empty subset of the full requirement can be used to achieve this requirement. For Class E, Not Safety Critical software, no test plans are required, but the project shall perform software testing. Class A_SC A_NSC B_SC B_NSC C_SC C_NSC D_SC D_NSC E_SC E_NSC F G H Applicable? P(C) P(C) Key: A_SC = Class A Software, Safety-Critical | A_NSC = Class A Software, Not Safety-Critical | ... | - Applicable | - Not Applicable Per NASA-GB-8719.13, NASA Software Safety Guidebook 276, "Testing serves several purposes: to find defects, to validate the system or an element of the system, and to verify functionality, performance, and safety requirements. The focus of testing is often on the verification and validation aspects. However, defect detection is probably the most important aspect of testing. While you cannot test quality into the software, you can certainly work to remove as many defects as possible." Following a plan helps ensure that all necessary and required testing tasks are performed. Development of the test plan provides the opportunity for stakeholders to give input and assist with the documentation and tailoring of the planned test activities for the project. Software testing has many levels including unit testing, integration testing, and system testing which can include functionality, performance, load, stress, safety, and acceptance testing. While unit testing is typically performed by the development team, some testing, such as integration, system or regression testing may be performed by a separate and/or independent test group. Keep in mind that formal testing, such as acceptance testing, is to be witnessed by an external organization, such as software assurance (see NASA-STD-8739.8, Software Assurance Standard 278, ). The following basic principles of testing come from NASA-GB-8719.13, NASA Software Safety Guidebook 276: NASA-GB-8719.13, NASA Software Safety Guidebook, 276 includes a chapter on testing with a focus on safety testing. Some general testing highlights of that chapter include: Other useful practices include: While NASA Centers typically have their own procedures and guidance, NASA-GB-8719.13, NASA Software Safety Guidebook, 276 lists and describes the following testing which needs to be considered when planning any software test effort: The following chart shows a basic flow for software testing activities from planning through maintenance. Several elements of this flow are addressed in related requirements in this Handbook (listed in the table at the end of this section). Tools which may be useful when performing software testing include the following, non-exhaustive list. Each project needs to evaluate and choose the appropriate tools for the testing to be performed for that project. Consult Center Process Asset Libraries (PALs) for Center-specific guidance and resources related to software testing. Additional guidance related to software testing, including specifics of plan, procedure, and report contents may be found in the following related requirements in this Handbook: Verify Implementation Evaluate Test Results Document Defects and Track Software Test Plan Test procedures Software Test Report Software testing is required regardless of project size. However, when time and resources are limited, areas of highest risk may be identified and priorities set to focus effort where the greatest benefit will be achieved with the available resources. (See Software QA and Testing Frequently-Asked-Questions 207 or NASA-GB-8719.13, NASA Software Safety Guidebook, 276 for suggested risk analysis considerations.) Tools to aid in compliance with this SWE, if any, may be found in the Tools Library in the NASA Engineering Network (NEN). NASA users find this in the Tools Library in the Software Processes Across NASA (SPAN) site of the Software Engineering Community in NEN. The list is informational only and does not represent an “approved tool list”, nor does it represent an endorsement of any particular tool. The purpose is to provide examples of tools being used across the Agency and to help projects and centers decide what tools to consider. The NASA Lessons Learned database contains the following lessons learned related to the importance of and potential issues related to software testing:
See edit history of this section
Post feedback on this section
1. Requirements
1.1 Notes
1.2 Applicability Across Classes
X - Applicable with details, read above for more | P(C) - P(Center), follow center requirements or procedures2. Rationale
3. Guidance
4. Small Projects
5. Resources
5.1 Tools
6. Lessons Learned
SWE-066 - Perform Testing
Web Resources
View this section on the websiteUnknown macro: {page-info}
"Scheduling testing phases is always an art, and depends on the expected quality of the software product. Relatively defect free software passes through testing within a minimal time frame. An inordinate amount of resources can be expended testing buggy software. Previous history, either of the development team or similar projects, can help determine how long testing will take. Some methods (such as error seeding and Halstead's defect metric) exist for estimating defect density (number of defects per unit of code) when historical information is not available." (NASA-GB-8719.13, NASA Software Safety Guidebook 276)