1. Requirements

3.9.2 The project manager shall satisfy the following conditions when a Commercial Off the Shelf (COTS), Government Off the Shelf (GOTS), Modified Off the Shelf (MOTS), or reused software component is to be acquired or used:

a. The requirements to be met by the software component are identified.

b. The software component includes documentation to fulfill its intended purpose (e.g., usage instructions).

c. Proprietary rights, usage rights, ownership, warranty, licensing rights, and transfer rights have been addressed.

d. Future support for the software product is planned and adequate for project needs.

e. The software component is verified and validated to the same level required to accept a similar developed software component for its intended use.

f. The project has a plan to perform periodic assessments of vendor reported defects to ensure the defects do not impact the selected software components.

1.1 Notes

The project responsible for procuring off-the-shelf software is responsible for documenting, prior to procurement, a plan for verifying and validating the software to the same level that would be required for a developed software component. The project ensures that the COTS, GOTS, MOTS, reused, and auto generated code software components and data meet the applicable requirements in this directive assigned to its software classification as shown in Appendix C. Open source requirements are in Section 3.15 [of the NPR].

1.2 Applicability Across Classes

If Class D software is safety critical, this requirement applies to the safety-critical aspects of the software.

Key:    - Applicable | - Not Applicable

2. Rationale

All software used on a project must meet the project requirements and be tested, verified and validated, including the incorporated Commercial Off the Shelf (COTS), Government Off the Shelf (GOTS), Modified Off the Shelf (MOTS), or reused software components. The project must know that each COTS, GOTS, MOTS, or reused software component meets NASA requirements; that all of the legal requirements for proprietary rights, usage rights, ownership, warranty, licensing rights, and transfer rights are understood and met in accordance with the project’s planned use; and that software future support for the software is planned.  To reduce the risk of failure, the project performs periodic assessments of vendor-reported defects to ensure the defects do not impact the selected software components.

3. Guidance

Several key concepts are addressed in this requirement for software that is used in a project, but which is not hand-generated. 

Identify Requirements

Identifying the requirements to be met by the software component allows the project to perform analysis to ensure the software component is adequate for the function it is intended to fulfill.  Identifying the requirements for the software component also allows the project to determine how the risk of using Commercial Off the Shelf (COTS), Government Off the Shelf (GOTS), Modified Off the Shelf (MOTS), or reused software components affects the overall risk posture of the software system. Identifying the requirements to be met by the COTS, GOTS, MOTS, or reused software components allows the project to perform testing to ensure that the COTS, GOTS, MOTS, or reused software component performs the requirement functions. Without requirements in place for the COTS, GOTS, MOTS, or reused software components the software functionality cannot be tested.

Obtain Documentation

The second concept is to ensure that the COTS, GOTS, MOTS, or reused software components include documentation to fulfill their intended purpose (e.g., usage instructions).  Usage instructions are important to help ensure the purpose, installation, and functions of the components are properly understood and used in the project.

Obtain Usage Rights

For COTS, GOTS, MOTS, or reused software components, projects need to be sure that any proprietary, usage, ownership, warranty, licensing rights, and transfer rights have been addressed to ensure that NASA has all the rights necessary to use the software component in the project.  It may be helpful to the project to work with procurement to determine the documents needed from the software developer.

Plan Future Support

Software components that are COTS, GOTS, MOTS, or reused software components may have limited lifetimes, so it is important to plan for future support of this software adequate for project needs.  As applicable, consider the following:

• Get a supplier agreement in place to deliver or escrow source code or a third party maintenance agreement.
• Ensure a risk mitigation plan is in place to cover the following cases:
  • Loss of supplier or third party support for the product.
  • Loss of maintenance for the product (or product version)
  • Loss of the product (e.g., license revoked, recall of product, etc.).
• Obtain an agreement that the project has access to defects discovered by the community of users. When available, the project can consider joining a product users group to obtain this information.
• Ensure a plan to provide adequate support is in place; the plan needs to include maintenance planning and the cost of maintenance.
• Document changes to the software management, development, operations, or maintenance plans that are affected by the use or incorporation of COTS, GOTS, MOTS, or reused software components.  

Ensure Fitness for Use

Commercial Off the Shelf (COTS), Government Off the Shelf (GOTS), Modified Off the Shelf (MOTS), or reused software components are required to be tested, verified and validated, to the level required to ensure its fitness for use in the intended application.  The software should be verified and validated, to the extent possible, in the same manner as software that is hand-generated using the project classification and criticality as the basis for the level of effort to be applied.

For COTS, GOTS, MOTS, or reused software components like commercial real-time operating systems, it is sufficient to test, in the project environment, the features being used to meet the software system’s requirements.  It is not necessary to test every claim made by the software.  On Class A projects, when the software test suites for the COTS, GOTS, MOTS, or reused software components are available, they are to be used when appropriate to address the intended environment of use, interfaces to the software system, as well as the requirements of the project.

Assess Vendor-Reported Defects

The project should plan to periodically assess vendor-reported defects in the COTS, GOTS, MOTS, or reused software components.  This assessment plan should include frequency of evaluations, likely within a short period of time following release of the vendor defect reports to users.  The plan should capture how the project will assess the impact of vendor-reported defects in the project’s environment.  This plan could refer back to procedures used to ensure the COTS, GOTS, MOTS, or reused software component’s fitness for use in the project environment and capture any additional activities necessary to ensure the defects do not impact system quality, reliability, performance, safety, etc..

Off-The-Shelf Software

The remaining guidance on off-the-shelf (OTS) software is broken down into elements as a function of the type of OTS software. The following is an index of the guidance elements:

3.1 COTS / GOTS Software

3.2 MOTS Software

3.3 Auto-generated Code

3.4 Embedded Software

Software reuse

Software reuse (either software acquired commercially or existing software from previous development activities) comes with a special set of concerns. Reuse of commercially-acquired software includes COTS, GOTS, and MOTS. Reuse of in-house software may include legacy or heritage software. Reused software often requires modification to be useable by the project. Modification may be extensive, or just require wrappers or glueware to make the software useable. Acquired and existing software must be evaluated during the process of selection to determine the effort required to bring it up to date. The basis for this evaluation is typically the criteria used for developing software as a new effort. The evaluation of the reused software requires ascertaining whether the quality of the software is sufficient for the intended application. The requirement statement for SWE-027 calls for five conditions to be satisfied In addition the note indicates six additional items to consider. The key item in the above listings is the need to assure the verification and validation (V&V) activity for the reused software is performed to the same level of confidence as would be required for the newly developed software component.

Software certification by outside sources

Outside certifications can be taken into account dependent on the intended NASA use of the software and the use of the software in the environment in which it was certified. Good engineering judgment has to be used in cases like this to determine the acceptable degree of use. An example: real-time operating system (RTOS) vendor certification data and testing can be used in conjunction with data certification done in the NASA software environment. Even though software has been determined "acceptable" by a regulatory entity [e.g., the Federal Aviation Administration (FAA)] good engineering judgment has to be used for these types of cases to determine the acceptable degree of use.

3.1 COTS/GOTS Software

Commercial Off the Shelf (COTS) software and  Government Off the Shelf (GOTS) software are unmodified, out of the box software solutions that can range in size from a portion of a software project to an entire software project. COTS/GOTS software can include software tools (e.g. word processor or spreadsheet applications), simulations (e.g. aeronautical and rocket simulations), and modeling tools (e.g., dynamics/thermal/electrical modeling tools).

If you are planning to use COTS/GOTS products, be sure to complete the checklist tables under the Tools section of this guidance. The purpose of these tables is to ensure that the table entries are considered in your software life cycle decisions from software acquisition through software maintenance.

If COTS/GOTS software is used for a portion of the software solution, the software requirements pertaining to that portion should be used in the testing, V&V of the COTS/GOTS software. For example, if a MIL-STD-1553 serial communications is the design solution for the project communications link requirements, and the COTS/GOTS software design solution is used along with the COTS/GOTS hardware design solution, then the project software requirements for the serial communications link should be used to test, verify and validate the COTS/GOTS MIL-STD-1553 software. Other functionality that might be present in the COTS/GOTS MIL-STD-1553 software may not be covered by the project requirements. This other functionality should be either disabled or determined to be safe by analysis and testing.

COTS software can range from simple software (e.g., handheld electronic device) to progressively more complicated software (e.g., launch vehicle control system software). A software criticality assessment (see SWE-133) and a risk assessment can be made to determine if the use of this software will result in an acceptable level of risk, even if unforeseen hazard(s) result in a failure. The results of these assessments can be used to set up the approach to using and verifying the COTS/GOTS software.

3.2 MOTS Software

As defined in Appendix A of NPR 7150.2, A.17:
"Modified Off-The-Shelf (MOTS) Software. When COTS, legacy/heritage software is reused, or heritage software is changed, the product is considered 'modified.'- The changes can include all or part of the software products and may involve additions, deletions, and specific alterations. An argument can be made that any alterations to the code and/or design of an off-the-shelf software component constitutes 'modification,' but the common usage allows for some percentage of change before the off-the-shelf software is declared to be MOTS software. This may include the changes to the application shell and/or glueware to add or protect against certain features and not to the off-the-shelf software system code directly. See the off-the-shelf [definition]."

In cases where legacy/heritage code is modified, MOTS is considered to be an efficient method to produce project software, especially if the legacy/heritage code is being used in the same application area of NASA. For example, Expendable Launch Vehicle simulations have been successfully modified to accommodate solid rocket boosters, payload release requirements, or other such changes. Further, if the "master" code has been designed with reuse in mind, such code becomes an efficient and effective method of producing quality code for succeeding projects.

An Independent Verification and Validation (IV&V) Facility report, "Software Reuse Study Report", April 29, 2005, examines in detail changes made on reused software. The conclusions are positive but caution against overestimating (underestimating) the extent and costs of reused software.

The Department of Defense (DoD) has had extensive experience in COTS and MOTS. A Lessons Learned item, Commercial Item Acquisition: Considerations and Lessons Learned, specifically includes lessons learned from MOTS. Concerns in these lessons learned included commercial software vendors attempting to modify existing commercial products, limiting the changes to "minor" modifications, and underestimating the extent and schedule impacts of testing of modified code.

Extreme caution should be exercised when attempting to purchase or modify COTS or GOTS code that was written for another application realm, or for which key documentation is missing (e.g., requirements, architecture, design, tests).

Engineering judgment and a consideration of possible impacts to the software development activity need to be used/made when determining that software is MOTS, legacy, or heritage.

3.3 Auto-generated Software

Auto-generated software is the result of translating a model of the system behavior created by software engineers into different languages such as C or Ada by the appropriate code generator. Changes are made by revising the model and code is generated from the revised model.

As is required for software developed without the use of a model or code generator auto-generated software is required to be verified and validated to the level required to ensure its fitness for use in the intended application.  Recommendations include subjecting the model to the same level of scrutiny and review as source code, ³⁶⁹ subjecting the auto-generated code to the same rigorous inspection, analysis, and test as hand-generated code, ²⁷⁶ and including system engineers and software engineers in joint reviews to identify misunderstood or unclear requirements.

There are many considerations to be reviewed when deciding whether using auto-generated software is the right choice for a project, including determining what is necessary to ensure future support for the generated software.  For recommended practices, considerations, and additional guidance, see topic 7.11 - Model Based Development and Auto-generated Code. 

3.4 Embedded Software

Embedded software applications written by/for NASA are commonly used by NASA for engineering software solutions. Embedded software is software specific to a particular application as opposed to general purpose software running on a desktop. Embedded software usually runs on custom computer hardware ("avionics"), often on a single chip.

Care must be taken when using vendor-supplied board support packages (BSPs) and hardware-specific software (drivers) which are typically supplied with off-the-shelf avionics systems. BSPs and drivers act as the software layer between the avionics hardware and the embedded software applications written by/for NASA. Most central processing unit (CPU) boards have BSPs provided by the board manufacturer, or third parties working with the board manufacturer. Driver software is provided for serial ports, universal serial bus (USB) ports, interrupt controllers, modems, printers, and many other hardware devices

BSPs and drivers are hardware dependent, often developed by third parties on hardware/software development tools which may not be accessible years later. Risk mitigation should include hardware-specific software, such as BSPs, software drivers, etc.

Many BSPs and drivers are provided by board manufacturers as binary code only, which could be an issue if the supplier is not available and BSP/driver errors are found. It is recommended that a project using BSPs/drivers maintain a configuration managed version of any BSPs with release dates and notes. Consult with avionics (hardware) engineers on the project to see what actions, if any,may be taken to configuration manage the BSPs/drivers.

Consideration should also be given to how BSP/driver software updates are to be handled, if and when they are made available, and how it will become known to the project that updates are available?

Vendor reports and user forums should be monitored from the time hardware and associated software are purchased through a reasonable time after deployment. Developers should monitor suppliers or user forums for bugs, workarounds, security changes, and other modifications to software that, if unknown, could derail a NASA project. Consider the following snippet from a user forum:

4. Small Projects

This requirement applies to all projects regardless of size.

5. Resources

5.1 Tools

Tools to aid in compliance with this SWE, if any, may be found in the Tools Library in the NASA Engineering Network (NEN).

NASA users find this in the Tools Library in the Software Processes Across NASA (SPAN) site of the Software Engineering Community in NEN.

The list is informational only and does not represent an “approved tool list”, nor does it represent an endorsement of any particular tool. The purpose is to provide examples of tools being used across the Agency and to help projects and centers decide what tools to consider.

6. Lessons Learned

The following information comes from the NASA Study on Flight Software Complexity listed in the reference section of this document:

"In 2007, a relatively new organization in DoD (the Software Engineering and System Assurance Deputy Directorate) reported their findings on software issues based on approximately 40 program reviews in the preceding 2½ years (Baldwin 2007). They found several software systemic issues that were significant contributors to poor program execution." Among the seven listed were the following on Commercial Off The Shelf (COTS):

• "Immature architectures, COTS integration, interoperability."

"Later, in partnership with the NDIA, they identified the seven top software issues that follow, drawn from a perspective of acquisition and oversight." Among the seven listed were the following on COTS:

• "Inadequate attention is given to total life cycle issues for COTS/NDI impacts on life cycle cost and risk."

"In partnership with the NDIA, they made seven corresponding top software recommendations." Among the seven listed were the following on COTS:

• "Improve and expand guidelines for addressing total life cycle COTS/NDI issues." ⁰⁴⁰

The following information is from Commercial Item Acquisition: Considerations and Lessons Learned July 14, 2000, Office of the Secretary of Defense:

This document is designed to assist DoD acquisition and supported commercial items. According to the introductory cover letter, "it provides an overview of the considerations inherent in such acquisitions and summarizes lessons learned from a wide variety of programs." Although it's written with the DoD acquirer in mind, it can provide useful information to you and assist you as we move down this increasingly significant path. ⁴²⁶

The NASA Lessons Learned database contains the following lessons learned related to the user of commercial, government, and legacy software:

• Lessons Learned From Flights of Off the Shelf Aviation Navigation Units on the Space Shuttle, GPS. Lesson Number 1370:  "The Shuttle Program selected off-the-shelf GPS and EGI units that met the requirements of the original customers. It was assumed that off-the-shelf units with proven design and performance would reduce acquisition costs and require minimal adaptation and minimal testing. However, the time, budget and resources needed to test and resolve firmware issues exceeded initial projections." ⁵⁵¹

• Lessons Learned Study Final Report for the Exploration Systems Mission Directorate, Langley Research Center; August 20, 2004. Lessons Learned Number 1838: "There has been an increasing interest in utilizing commercially available hardware and software as portions of space flight systems and their supporting infrastructure. Experience has shown that this is a very satisfactory approach for some items, and a major mistake for others. In general, COTS [products] should not be used as part of any critical systems [but see the recommendation later in this Lesson Learned] because of the generally lower level of engineering and product assurance used in their manufacture and test. In those situations where COTS [software] has been applied to flight systems, such as the laptop computers utilized as control interfaces on [International Space Station] (ISS), the cost of modifying and testing the hardware/software to meet flight requirements has far exceeded expectations, potentially defeating the reason for selecting COTS products in the first place. In other cases, such as the [Checkout Launch Control System] (CLCS) project at JSC, the cost of maintaining the commercial software had not been adequately analyzed and drove the project's recurring costs outside the acceptable range.
  Recommendation: Ensure that candidate COTS products are thoroughly analyzed for technical deficiencies and life-cycle cost implications before levying them on the program.
  
  • COTS systems have potential to reduce system costs, but only if all of their characteristics are considered beforehand and included in the planned application. (Standards)
     
  • COTS systems that look good on paper may not scale well to NASA needs for legitimate reasons. These include sustaining engineering/update cycle/recertification costs, scaling effects, dependence on third party services and products. Need to assure that a life-cycle cost has been considered correctly. (Headquarters - CLCS) ⁴²⁴
• ADEOS-II NASA Ground Network (NGN) Development and Early Operations – Central/Standard Autonomous File Server (CSAFS/SAFS) Lessons Learned. Lesson Number 1346: "The purpose of the Standard Autonomous File Server (SAFS) is to provide automated management of large data files without interfering with the assets involved in the acquisition of the data. It operates as a stand-alone solution, monitoring itself, and providing an automated level of fail-over processing to enhance reliability. The successful integration of COTS products into the SAFS system has been key to its becoming accepted as a NASA standard resource for file distribution, and leading to its nomination for NASA's Software of the Year Award in 1999."
   
  Lessons Learned:
  "Match COTS tools to project requirements. Deciding to use a COTS product as the basis of system software design is potentially risky, but the potential benefits include quicker delivery, less cost, and more reliability in the final product. The following lessons were learned in the definition phase of the SAFS/CSAFS development.
  
  • "Use COTS products and re-use previously developed internal products.
  • "Create a prioritized list of desired COTS features.
  • "Talk with local experts having experience in similar areas.
  • "Conduct frequent peer and design reviews.
  • "Obtain demonstration [evaluation] versions of COTS products.
  • "Obtain customer references from vendors.
  • "Select a product appropriately sized for your application.
  • "Choose a product closely aligned with your project's requirements.
  • "Select a vendor whose size will permit a working relationship.
  • "Use vendor tutorials, documentation, and vendor contacts during COTS evaluation period."
     
    "Test and prototype COTS products in the lab. The prototyping and test phase of the COTS evaluation allows problems to be identified as the system design matures. These problems can be mitigated (often with the help and cooperation of the COTS vendor) well before the field-testing phase at which time it may be too costly or impossible to retrofit a solution. The following lessons were learned in the prototyping and test phase of the SAFS/CSAFS development:
     
  • "Prototype your systems hardware and software in a lab setting as similar to the field environment as possible.
    
    • "Simulate how the product will work on various customer platforms.
    • "Model the field operations.
    • "Develop in stages with ongoing integration and testing."
       
  • "Pass pertinent information on to your customers.
  • "Accommodate your customers, where possible, by building in alternative options.
  • "Don't approve all requests for additional options by customers or new projects that come on line.
  • "Select the best COTS components for product performance even if they are from multiple vendors.
  • "Consider the expansion capability of any COTS product.
  • "Determine if the vendor's support is adequate for your requirements.
     
    "Install, operate and maintain the COTS field and lab components. The following lessons were learned in the installation and operation phase of the SAFS/CSAFS development:
     
  • "Personally perform on-site installations whenever possible.
  • "Have support/maintenance contracts for hardware and software through development, deployment, and first year of operation.
  • "Create visual representations of system interactions where possible.
  • "Obtain feedback from end users.
  • "Maintain the prototype system after deployment.
  • "Select COTS products with the ability to do internal logging." ⁵⁵⁰

• MER Spirit Flash Memory Anomaly (2004).  Lesson Number 1483: "Shortly after the commencement of science activities on Mars, the Mars Exploration Rover (MER) lost the ability to execute any task that requested memory from the flight computer. The cause was incorrect configuration parameters in two operating system software modules that control the storage of files in system memory and flash memory. Seven recommendations cover enforcing design guidelines for COTS software, verifying assumptions about software behavior, maintaining a list of lower priority action items, testing flight software internal functions, creating a comprehensive suite of tests and automated analysis tools, providing downlinked data on system resources, and avoiding the problematic file system and complex directory structure."
• Recommendations:
  
  • "Enforce the project-specific design guidelines for COTS software, as well as for NASA-developed software. Assure that the flight software development team reviews the basic logic and functions of commercial off-the-shelf (COTS) software, with briefings and participation by the vendor.
  • "Verify assumptions regarding the expected behavior of software modules. Do not use a module without detailed peer review, and assure that all design and test issues are addressed.
  • "Where the software development schedule forestalls completion of lower priority action items, maintain a list of incomplete items that require resolution before final configuration of the flight software.
  • "Place high priority on completing tests to verify the execution of flight software internal functions.
  • "Early in the software development process, create a comprehensive suite of tests and automated analysis tools.
  • "Ensure that reporting flight computer related resource usage is included.
  • "Ensure that the flight software downlinks data on system resources (such as the free system memory) so that the actual and expected behavior of the system can be compared.
  • "For future missions, implement a more robust version of the dosFsLib module, and/or use a different type of file system and a less complex directory structure." ⁵⁵⁷.
     

• International Space Station Lessons Learned as Applied to Exploration, KSC, July 22, 2009:(23-Lesson): Use Commercial Off-the-Shelf Products Where Possible.
  • An effective strategy in the ISS program was to simplify designs by utilizing commercial off-the-shelf (COTS) hardware and software products for non-safety, non-critical applications.
  • Application to Exploration: Use of COTS products should be encouraged whenever practical in exploration programs. ⁴²⁵