See edit history of this section
Post feedback on this section
- 1. The Requirement
- 2. Rationale
- 3. Guidance
- 4. Small Projects
- 5. Resources
- 6. Lessons Learned
- 7. Software Assurance
1. Requirements
4.5.14 The project manager shall test embedded COTS, GOTS, MOTS, OSS, or reused software components to the same level required to accept a custom developed software component for its intended use.
1.1 Notes
NPR 7150.2, NASA Software Engineering Requirements, does not include any notes for this requirement.
1.2 History
1.3 Applicability Across Classes
Class A B C D E F Applicable?
Key: - Applicable | - Not Applicable
2. Rationale
Software testing is required to ensure that the software meets the agreed requirements and design, the application works as expected, the application doesn’t contain serious bugs and the software meets its intended use as per user expectations. Testing of embedded COTS, GOTS, MOTS, Open Source Software (OSS), or reused software component should be at the same level required to accept a custom-developed software component for its intended use.
3. Guidance
Commercial Off the Shelf (COTS), Government Off the Shelf (GOTS), Modified Off the Shelf (MOTS), Open Source Software (OSS), or reused software components are required to be tested, verified and validated, to the level required to ensure its fitness for use in the intended application. The software should be verified and validated, to the extent possible, in the same manner as software that is hand-generated using the project classification and criticality as the basis for the level of effort to be applied.
For COTS, GOTS, MOTS, or reused software components like commercial real-time operating systems, it is sufficient to test, in the project environment, the features being used to meet the software system’s requirements. It is not necessary to test every claim made by the software. On Class A projects, when the software test suites for the COTS, GOTS, MOTS, or reused software components are available, they are to be used when appropriate to address the intended environment of use, interfaces to the software system, as well as the requirements of the project.
Off-The-Shelf Software
The remaining guidance on off-the-shelf (OTS) software is broken down into elements as a function of the type of OTS software. See also SWE-027 - Use of Commercial, Government, and Legacy Software.
3.1 COTS/GOTS Software
Commercial Off the Shelf (COTS) software and Government Off the Shelf (GOTS) software are unmodified, out of the box software solutions that can range in size from a portion of a software project to an entire software project. COTS/GOTS software can include software tools (e.g. word processor or spreadsheet applications), simulations (e.g. aeronautical and rocket simulations), and modeling tools (e.g., dynamics/thermal/electrical modeling tools).
If COTS/GOTS software is used for a portion of the software solution, the software requirements about that portion should be used in the testing, V&V of the COTS/GOTS software. For example, if MIL-STD-1553 668 serial communication is the design solution for the project communications link requirements, and the COTS/GOTS software design solution is used along with the COTS/GOTS hardware design solution, then the project software requirements for the serial communications link should be used to test, verify and validate the COTS/GOTS MIL-STD-1553 software. Other functionality that might be present in the COTS/GOTS MIL-STD-1553 software may not be covered by the project requirements. This other functionality should be either disabled or determined to be safe by analysis and testing.
See also Topic 8.08 - COTS Software Safety Considerations.
3.2 MOTS Software
As defined in Appendix A of NPR 7150.2,
Modified Off-the-Shelf Software (MOTS).
"When COTS or legacy and heritage software is reused, or heritage software is changed, the product is considered "modified." The changes can include all or part of the software products and may involve additions, deletions, and specific alterations. An argument can be made that any alterations to the code and design of an off-the-shelf software component constitute "modification," but the common usage allows for some percentage (less than 5 percent of the code changes) of change before the off-the-shelf software is declared to be modified off-the-shelf (MOTS) software. Modified Off-the-Shelf Software may include the changes to the application shell or glueware to add or protect against certain features and not to the off-the-shelf software system code directly." 083
In cases where legacy/heritage code is modified, MOTS is considered to be an efficient method to produce project software, especially if the legacy/heritage code is being used in the same application area of NASA. For example, Expendable Launch Vehicle simulations have been successfully modified to accommodate solid rocket boosters, payload release requirements, or other such changes. Further, if the "master" code has been designed with reuse in mind, such code becomes an efficient and effective method of producing quality code for succeeding projects.
3.3 Open Source Software
Open Source Software (OSS) is software with source code that anyone can inspect, modify, and enhance. Open Source can bring numerous benefits to NASA software efforts, including increased software quality, reduced development costs, faster development cycles, and reduced barriers to public-private collaboration through new opportunities to commercialize NASA technology.3.4 Embedded Software
Embedded software applications written by/for NASA are commonly used by NASA for engineering software solutions. Embedded software is software specific to a particular application as opposed to general-purpose software running on a desktop. Embedded software usually runs on custom computer hardware ("avionics"), often on a single chip.
3.5 Software reuse
The reuse of commercially acquired software includes COTS, GOTS, and MOTS. Reuse of in-house software may include legacy or heritage software. Reused software often requires modification to be used by the project. Modification may be extensive, or just require wrappers or glueware to make the software useable. Assure the verification and validation (V&V) activity for the reused software is performed to the same level of confidence as would be required for the newly developed software component.
3.6 Additional Guidance
Additional guidance related to this requirement may be found in the following materials in this Handbook:
Related Links |
---|
3.7 Center Process Asset Libraries
SPAN - Software Processes Across NASA
SPAN contains links to Center managed Process Asset Libraries. Consult these Process Asset Libraries (PALs) for Center-specific guidance including processes, forms, checklists, training, and templates related to Software Development. See SPAN in the Software Engineering Community of NEN. Available to NASA only. https://nen.nasa.gov/web/software/wiki 197
See the following link(s) in SPAN for process assets from contributing Centers (NASA Only).
SPAN Links |
---|
4. Small Projects
Small projects may choose to use non-custom developed software but need to understand the limitations and assumptions that went into the development of that software. Reuse of software has different challenges, but all projects should ask for test suites from previously developed software vendors to ensure that the integrated components still operate the same. If the software being selected is used for multiple small project campaigns (i.e., different science collected on sounding rockets), the testing costs can be performed or shared between these projects, especially if using the same CSCI.
5. Resources
5.1 References
- (SWEREF-083) NPR 7150.2D, Effective Date: March 08, 2022, Expiration Date: March 08, 2027 https://nodis3.gsfc.nasa.gov/displayDir.cfm?t=NPR&c=7150&s=2D Contains link to full text copy in PDF format. Search for "SWEREF-083" for links to old NPR7150.2 copies.
- (SWEREF-197) Software Processes Across NASA (SPAN) web site in NEN SPAN is a compendium of Processes, Procedures, Job Aids, Examples and other recommended best practices.
- (SWEREF-668) MIL-STD-1553B, published in 1978,
5.2 Tools
NASA users find this in the Tools Library in the Software Processes Across NASA (SPAN) site of the Software Engineering Community in NEN.
The list is informational only and does not represent an “approved tool list”, nor does it represent an endorsement of any particular tool. The purpose is to provide examples of tools being used across the Agency and to help projects and centers decide what tools to consider.
6. Lessons Learned
6.1 NASA Lessons Learned
No Lessons Learned have currently been identified for this requirement.
6.2 Other Lessons Learned
No other Lessons Learned have currently been identified for this requirement.
7. Software Assurance
7.1 Tasking for Software Assurance
1. Confirm that the project is testing COTS, GOTS, MOTS, OSS, or reused software components to the same level as developed software for its intended use.
7.2 Software Assurance Products
- None at this time.
Objective Evidence
- Software test reports.
- Software test procedures.
7.3 Metrics
- # of Non-Conformances identified in embedded COTS, GOT, MOTS, OSS, or reused components in ground or flight software vs. # of Non-Conformances successfully closed
- # of tests completed vs. total # of tests
- # of Non-Conformances identified during each testing phase (Open, Closed, Severity)
- # of tests executed vs. # of tests completed
See also Topic 8.18 - SA Suggested Metrics.
7.4 Guidance
For software assurance to confirm that the COTS, GOTS, MOTS, OSS, or reused code is being tested to the same level as developed components, they must know what capabilities and requirements are being satisfied by each of the COTS, GOTS, MOTS, and OSS components. When those requirements and capabilities are identified, software assurance will confirm that each of those capabilities/requirements has been included in the test procedures and test cases.
Then software assurance will confirm that this set of tests has been run and satisfactorily completed, either by witnessing the tests or by examining the test reports. If changes have been made to the GOTS, MOTS, OSS, or reused software, confirm that these changes have been adequately tested, including regression testing so that all requirements are still satisfied with no unintended consequences. If this software is satisfying any safety-critical requirements, a full set of regression tests needs to be run.
Software assurance signs off that all tests have been completed.
See also Topic 8.08 - COTS Software Safety Considerations.
7.5 Additional Guidance
Additional guidance related to this requirement may be found in the following materials in this Handbook: