Several key concepts are addressed in this requirement for software that is used in a project, but which is not hand-generated.
Identifying the requirements to be met by the software component allows the project to perform analysis to ensure the software component is adequate for the function it is intended to fulfill. Identifying the requirements for the software component also allows the project to determine how the risk of using Commercial Off the Shelf (COTS), Government Off the Shelf (GOTS), Modified Off the Shelf (MOTS), or reused software components affects the overall risk posture of the software system. Identifying the requirements to be met by the COTS, GOTS, MOTS, or reused software components allows the project to perform testing to ensure that the COTS, GOTS, MOTS, or reused software component performs the requirement functions. Without requirements in place for the COTS, GOTS, MOTS, or reused software components the software functionality cannot be tested.
The second concept is to ensure that the COTS, GOTS, MOTS, or reused software components include documentation to fulfill their intended purpose (e.g., usage instructions). Usage instructions are important to help ensure the purpose, installation, and functions of the components are properly understood and used in the project.
Obtain Usage Rights
For COTS, GOTS, MOTS, or reused software components, projects need to be sure that any proprietary, usage, ownership, warranty, licensing rights, and transfer rights have been addressed to ensure that NASA has all the rights necessary to use the software component in the project. It may be helpful to the project to work with procurement to determine the documents needed from the software developer.
Plan Future Support
Software components that are COTS, GOTS, MOTS, or reused software components may have limited lifetimes, so it is important to plan for future support of this software adequate for project needs. As applicable, consider the following:
- Get a supplier agreement in place to deliver or escrow source code or a third party maintenance agreement.
- Ensure a risk mitigation plan is in place to cover the following cases:
- Loss of supplier or third party support for the product.
- Loss of maintenance for the product (or product version)
- Loss of the product (e.g., license revoked, recall of product, etc.).
- Obtain an agreement that the project has access to defects discovered by the community of users. When available, the project can consider joining a product users group to obtain this information.
- Ensure a plan to provide adequate support is in place; the plan needs to include maintenance planning and the cost of maintenance.
- Document changes to the software management, development, operations, or maintenance plans that are affected by the use or incorporation of COTS, GOTS, MOTS, or reused software components.
Ensure Fitness for Use
Commercial Off the Shelf (COTS), Government Off the Shelf (GOTS), Modified Off the Shelf (MOTS), or reused software components are required to be tested, verified and validated, to the level required to ensure its fitness for use in the intended application. The software should be verified and validated, to the extent possible, in the same manner as software that is hand-generated using the project classification and criticality as the basis for the level of effort to be applied.
For COTS, GOTS, MOTS, or reused software components like commercial real-time operating systems, it is sufficient to test, in the project environment, the features being used to meet the software system’s requirements. It is not necessary to test every claim made by the software. On Class A projects, when the software test suites for the COTS, GOTS, MOTS, or reused software components are available, they are to be used when appropriate to address the intended environment of use, interfaces to the software system, as well as the requirements of the project.
Assess Vendor-Reported Defects
The project should plan to periodically assess vendor-reported defects in the COTS, GOTS, MOTS, or reused software components. This assessment plan should include frequency of evaluations, likely within a short period of time following release of the vendor defect reports to users. The plan should capture how the project will assess the impact of vendor-reported defects in the project’s environment. This plan could refer back to procedures used to ensure the COTS, GOTS, MOTS, or reused software component’s fitness for use in the project environment and capture any additional activities necessary to ensure the defects do not impact system quality, reliability, performance, safety, etc..
The remaining guidance on off-the-shelf (OTS) software is broken down into elements as a function of the type of OTS software. The following is an index of the guidance elements:
3.1 COTS / GOTS Software
3.2 MOTS Software
3.3 Auto-generated Code
3.4 Embedded Software
Software reuse (either software acquired commercially or existing software from previous development activities) comes with a special set of concerns. Reuse of commercially-acquired software includes COTS, GOTS, and MOTS. Reuse of in-house software may include legacy or heritage software. Reused software often requires modification to be useable by the project. Modification may be extensive, or just require wrappers or glueware to make the software useable. Acquired and existing software must be evaluated during the process of selection to determine the effort required to bring it up to date. The basis for this evaluation is typically the criteria used for developing software as a new effort. The evaluation of the reused software requires ascertaining whether the quality of the software is sufficient for the intended application. The requirement statement for SWE-027 calls for five conditions to be satisfied In addition the note indicates six additional items to consider. The key item in the above listings is the need to assure the verification and validation (V&V) activity for the reused software is performed to the same level of confidence as would be required for the newly developed software component.
Software certification by outside sources
Outside certifications can be taken into account dependent on the intended NASA use of the software and the use of the software in the environment in which it was certified. Good engineering judgment has to be used in cases like this to determine the acceptable degree of use. An example: real-time operating system (RTOS) vendor certification data and testing can be used in conjunction with data certification done in the NASA software environment. Even though software has been determined "acceptable" by a regulatory entity [e.g., the Federal Aviation Administration (FAA)] good engineering judgment has to be used for these types of cases to determine the acceptable degree of use.
3.1 COTS/GOTS Software
Commercial Off the Shelf (COTS) software and Government Off the Shelf (GOTS) software are unmodified, out of the box software solutions that can range in size from a portion of a software project to an entire software project. COTS/GOTS software can include software tools (e.g. word processor or spreadsheet applications), simulations (e.g. aeronautical and rocket simulations), and modeling tools (e.g., dynamics/thermal/electrical modeling tools).
If you are planning to use COTS/GOTS products, be sure to complete the checklist tables under the Tools section of this guidance. The purpose of these tables is to ensure that the table entries are considered in your software life cycle decisions from software acquisition through software maintenance.
If COTS/GOTS software is used for a portion of the software solution, the software requirements pertaining to that portion should be used in the testing, V&V of the COTS/GOTS software. For example, if a MIL-STD-1553 serial communications is the design solution for the project communications link requirements, and the COTS/GOTS software design solution is used along with the COTS/GOTS hardware design solution, then the project software requirements for the serial communications link should be used to test, verify and validate the COTS/GOTS MIL-STD-1553 software. Other functionality that might be present in the COTS/GOTS MIL-STD-1553 software may not be covered by the project requirements. This other functionality should be either disabled or determined to be safe by analysis and testing.
COTS software can range from simple software (e.g., handheld electronic device) to progressively more complicated software (e.g., launch vehicle control system software). A software criticality assessment (see SWE-133) and a risk assessment can be made to determine if the use of this software will result in an acceptable level of risk, even if unforeseen hazard(s) result in a failure. The results of these assessments can be used to set up the approach to using and verifying the COTS/GOTS software.
3.2 MOTS Software
As defined in Appendix A of NPR 7150.2, A.17:
"Modified Off-The-Shelf (MOTS) Software. When COTS, legacy/heritage software is reused, or heritage software is changed, the product is considered 'modified.'- The changes can include all or part of the software products and may involve additions, deletions, and specific alterations. An argument can be made that any alterations to the code and/or design of an off-the-shelf software component constitutes 'modification,' but the common usage allows for some percentage of change before the off-the-shelf software is declared to be MOTS software. This may include the changes to the application shell and/or glueware to add or protect against certain features and not to the off-the-shelf software system code directly. See the off-the-shelf [definition]."
In cases where legacy/heritage code is modified, MOTS is considered to be an efficient method to produce project software, especially if the legacy/heritage code is being used in the same application area of NASA. For example, Expendable Launch Vehicle simulations have been successfully modified to accommodate solid rocket boosters, payload release requirements, or other such changes. Further, if the "master" code has been designed with reuse in mind, such code becomes an efficient and effective method of producing quality code for succeeding projects.
An Independent Verification and Validation (IV&V) Facility report, "Software Reuse Study Report", April 29, 2005, examines in detail changes made on reused software. The conclusions are positive but caution against overestimating (underestimating) the extent and costs of reused software.
The Department of Defense (DoD) has had extensive experience in COTS and MOTS. A Lessons Learned item, Commercial Item Acquisition: Considerations and Lessons Learned, specifically includes lessons learned from MOTS. Concerns in these lessons learned included commercial software vendors attempting to modify existing commercial products, limiting the changes to "minor" modifications, and underestimating the extent and schedule impacts of testing of modified code.
Extreme caution should be exercised when attempting to purchase or modify COTS or GOTS code that was written for another application realm, or for which key documentation is missing (e.g., requirements, architecture, design, tests).
Engineering judgment and a consideration of possible impacts to the software development activity need to be used/made when determining that software is MOTS, legacy, or heritage.
3.3 Auto-generated Software
Auto-generated software is the result of translating a model of the system behavior created by software engineers into different languages such as C or Ada by the appropriate code generator. Changes are made by revising the model and code is generated from the revised model.
As is required for software developed without the use of a model or code generator auto-generated software is required to be verified and validated to the level required to ensure its fitness for use in the intended application. Recommendations include subjecting the model to the same level of scrutiny and review as source code, subjecting the auto-generated code to the same rigorous inspection, analysis, and test as hand-generated code, and including system engineers and software engineers in joint reviews to identify misunderstood or unclear requirements.
There are many considerations to be reviewed when deciding whether using auto-generated software is the right choice for a project, including determining what is necessary to ensure future support for the generated software. For recommended practices, considerations, and additional guidance, see Topic 7.11 – Model Based Development and Auto-generated Code.
3.4 Embedded Software
Embedded software applications written by/for NASA are commonly used by NASA for engineering software solutions. Embedded software is software specific to a particular application as opposed to general purpose software running on a desktop. Embedded software usually runs on custom computer hardware ("avionics"), often on a single chip.
Care must be taken when using vendor-supplied board support packages (BSPs) and hardware-specific software (drivers) which are typically supplied with off-the-shelf avionics systems. BSPs and drivers act as the software layer between the avionics hardware and the embedded software applications written by/for NASA. Most central processing unit (CPU) boards have BSPs provided by the board manufacturer, or third parties working with the board manufacturer. Driver software is provided for serial ports, universal serial bus (USB) ports, interrupt controllers, modems, printers, and many other hardware devices
BSPs and drivers are hardware dependent, often developed by third parties on hardware/software development tools which may not be accessible years later. Risk mitigation should include hardware-specific software, such as BSPs, software drivers, etc.
Many BSPs and drivers are provided by board manufacturers as binary code only, which could be an issue if the supplier is not available and BSP/driver errors are found. It is recommended that a project using BSPs/drivers maintain a configuration managed version of any BSPs with release dates and notes. Consult with avionics (hardware) engineers on the project to see what actions, if any,may be taken to configuration manage the BSPs/drivers.
Consideration should also be given to how BSP/driver software updates are to be handled, if and when they are made available, and how it will become known to the project that updates are available?
Vendor reports and user forums should be monitored from the time hardware and associated software are purchased through a reasonable time after deployment. Developers should monitor suppliers or user forums for bugs, workarounds, security changes, and other modifications to software that, if unknown, could derail a NASA project. Consider the following snippet from a user forum:
"Manufacturer Pt. No." motherboard embedded complex electronics contains malware
A "Manufacturer" support forum identifies "manufacturer's product" motherboards which contain harmful code. The embedded complex electronics for server management on some motherboards may contain malicious code. There is no impact on either new servers or non-Windows based servers. No further information is available regarding the malware, malware mitigation, the serial number of motherboards affected, nor the source.