- The requirements that are to be met by the software component are identified.
- The software component includes documentation to fulfill its intended purpose (e.g., usage instructions).
- Proprietary, usage, ownership, warranty, licensing rights, and transfer rights have been addressed.
- Future support for the software product is planned.
- The software component is verified and validated to the same level of confidence as would be required of the developed software component.
The project responsible for procuring off-the-shelf software is responsible for documenting, prior to procurement, a plan for verifying and validating the off-the-shelf software to the same level of confidence that would be needed for an equivalent class of software if obtained through a "development" process. The project ensures that the COTS, GOTS, MOTS, reused, and open source software components and data meet the applicable requirements in this NPR [NPR 7150.2, NASA Software Engineering Requirements] assigned to its software classification as shown in Appendix D.
Note: For these types of software components consider the following:
- Supplier agreement to deliver or escrow source code or third party maintenance agreement is in place.
- A risk mitigation plan to cover the following cases is available:
- Loss of supplier or third party support for the product.
- Loss of maintenance for the product (or product version).
- Loss of the product (e.g., license revoked, recall of product, etc.).
- An Agreement that the project has access to defects discovered by the community of users has been obtained. When available, the project can consider joining a product users group to obtain this information.
- A plan to provide adequate support is in place; the plan needs to include maintenance planning and the cost of maintenance.
- Documentation changes to the software management, development, operations, or maintenance plans that are affected by the use or incorporation of COTS, GOTS, MOTS, reused, and legacy/heritage software.
- Open source software licenses review by the Center Counsel.
1.2 Applicability Across Classes
Class G is labeled with "P (Center)." This means that an approved Center-defined process which meets a non-empty subset of the full requirement can be used to achieve this requirement.
This requirement exists in NPR 7150.2 because some software (e.g. COTS, GOTS) is purchased with no direct NASA or NASA contractor software engineering involvement in software development. Projects using this type of COTS/GOTS software must know that the acquisition and maintenance of the software is expected to meet NASA requirements as spelled out in this section of NPR 7150.2 (see Topic 7.3 - Acquisition Guidance).
This requirement also exists in NPR 7150.2 because some software, whether purchased as COTS, GOTS or developed/modified in house may contain Open Source Software. If Open Source Software exists within the project software, there may be implications on how the software can be used in the future, including internal/external releases or reuse of the software.
Finally, this requirement exists in NPR 7150.2 because some software may be heritage (or legacy) software, or developed/purchased before current software engineering processes were in place.
Software reuse (either software acquired commercially or existing software from previous development activities) comes with a special set of concerns. Reuse of commercially-acquired software includes COTS, GOTS, and MOTS. Reuse of in-house software may include legacy or heritage software. Open Source Software is also reused software. Reused software often requires modification to be useable by the project. Modification may be extensive, or just require wrappers or glueware to make the software useable. Acquired and existing software must be evaluated during the process of selection to determine the effort required to bring it up to date. The basis for this evaluation is typically the criteria used for developing software as a new effort. The evaluation of the reused software requires ascertaining whether the quality of the software is sufficient for the intended application. The requirement statement for SWE-027 calls for five conditions to be satisfied In addition the note indicates six additional items to consider. The key item in the above listings is the need to assure the verification and validation (V&V) activity for the reused software is performed to the same level of confidence as would be required for the newly developed software component.
Software certification by outside sources:
3.1 COTS/GOTS Software
COTS (commercial off-the-shelf) software and GOTS (government off-the-shelf) software are unmodified, out of the box software solutions that can range in size from a portion of a software project to an entire software project. COTS/GOTS software can include software tools (e.g. word processor or spreadsheet applications), simulations (e.g. aeronautical and rocket simulations), and modeling tools (e.g., dynamics/thermal/electrical modeling tools).
If you are planning to use COTS/GOTS products, be sure to complete the checklist tables under the Tools section. The purpose of these tables is to ensure that the table entries are considered in your software life cycle decisions from software acquisition through software maintenance.
If COTS/GOTS software is used for a portion of the software solution, the software requirements pertaining to that portion should be used in the testing, V&V of the COTS/GOTS software. For example, if a MIL-STD-1553 serial communications is the design solution for the project communications link requirements, and the COTS/GOTS software design solution is used along with the COTS/GOTS hardware design solution, then the project software requirements for the serial communications link should be used to test, verify and validate the COTS/GOTS MIL-STD-1553 software. Other functionality that might be present in the COTS/GOTS MIL-STD-1553 software may not be covered by the project requirements. This other functionality should be either disabled or determined to be safe by analysis and testing.
COTS software can range from simple software (e.g., handheld electronic device) to progressively more complicated software (e.g., launch vehicle control system software). A software criticality assessment (see SWE-133) and a risk assessment can be made to determine if the use of this software will result in an acceptable level of risk, even if unforeseen hazard(s) result in a failure. The results of these assessments can be used to set up the approach to using and verifying the COTS/GOTS software.
3.2 MOTS Software
As defined in Appendix A of NPR 7150.2, A.18:
"Modified Off-The-Shelf (MOTS) Software. When COTS, legacy/heritage software is reused, or heritage software is changed, the product is considered 'modified.'- The changes can include all or part of the software products and may involve additions, deletions, and specific alterations. An argument can be made that any alterations to the code and/or design of an off-the-shelf software component constitutes 'modification,' but the common usage allows for some percentage of change before the off-the-shelf software is declared to be MOTS software. This may include the changes to the application shell and/or glueware to add or protect against certain features and not to the off-the-shelf software system code directly. See the off-the-shelf [definition]."
In cases where legacy/heritage code is modified, MOTS is considered to be an efficient method to produce project software, especially if the legacy/heritage code is being used in the same application area of NASA. For example, Expendable Launch Vehicle simulations have been successfully modified to accommodate solid rocket boosters, payload release requirements, or other such changes. Further, if the "master" code has been designed with reuse in mind, such code becomes an efficient and effective method of producing quality code for succeeding projects.
An Independent Verification and Validation (IV&V) Facility report, "Software Reuse Study Report", April 29, 2005, examines in detail changes made on reused software. The conclusions are positive but caution against overestimating (underestimating) the extent and costs of reused software.
The DoD has had extensive experience in COTS/GOTS and MOTS. A Lessons Learned item, Commercial Item Acquisition: Considerations and Lessons Learned, specifically includes lessons learned from MOTS. Concerns in these lessons learned included commercial software vendors attempting to modify existing commercial products, limiting the changes to "minor" modifications, and underestimating the extent and schedule impacts of testing of modified code.
Extreme caution should be exercised when attempting to purchase or modify COTS or GOTS code that was written for another application realm, or for which key documentation is missing (e.g., requirements, architecture, design, tests).
Engineering judgement and a consideration of possible impacts to the software development activity need to be used/made when determining that software is MOTS, legacy, or heritage.
3.3 Legacy/Heritage Code
"Legacy" and "heritage" code will be used interchangeably here to identify code that may have been produced before modern software development processes were put into place. Reused code may in many cases be considered legacy/heritage code. Legacy code can range in size from small units of code (e.g., subroutines) to large, complete software systems (e.g., Atlas-Centaur Expendable Launch Vehicle Simulation).
It may be desirable to maintain legacy code largely intact due to one or more of the following factors:
- The code may have a history of successful applications over many runs.
- No new software errors have been found in the code in some time and it has been reliable through many years of use.
- The cost of upgrading the legacy code (e.g., a new software development) may be uneconomical or unaffordable in terms of time or funding.
- Upgrading the legacy code could add new software errors.
- Software personnel are familiar with the legacy code.
- Safety reviews have been conducted on the legacy code under similar applications.
On the other hand, it may be desirable to replace legacy code due to one or more of the following factors:
- No active civil servants or contractors are familiar with the code or its operation.
- One or more of the following documents are missing: architecture, requirements, traceability, design, source code, unit through integration test cases, validation results, user operational manuals, non-conformances, waivers, or coding standards.
- Lack of conditions for the installation of the software or use of the software or software development environment.
- No safety review has been done on the new code in its old or new operational environment.
- The legacy code may contain Open Source Software with questionable license terms or rights.
- The source code language compilers may be years out of date or even inaccessible.
- Emulators may not be available.
- Maintenance responsibility is unknown.
- Legacy code may operate on out-of-date or unavailable operating systems.
- Unknown IP, licensing, exportability constraints, if any.
Determining which path to follow (keep or replace legacy code) is largely a cost-risk analysis. Further guidance on this facet of legacy code will be provided in future iterations of this Electronic Handbook.
If the decision is made to maintain the use of the legacy code, it is recommended that incremental improvements be made as the code is used. Don't wait until all the experts retire! Specifically,
- Requirements should be documented if not already available.
- Create V&V documents based on testing against any vendor documentation, including user's manuals.
- Start configuration management on the reused code.
- Software architecture and design should be documented, if not already available.
- Test cases should be documented.
- Software debugging and error reporting should be documented.
- Software Assurance and Safety personnel should review the legacy code and documentation.
- All documentation, test results, maintenance history, and other such documents associated with legacy code should be gathered and stored, if it is anticipated the code will be reused.
Engineering judgement and a consideration of possible impacts to the software development activity need to be used/made when determining that software is MOTS, legacy, or heritage.
3.4 Open Source Software
Open Source Software is considered a form of off-the-shelf software. Even if most of the software on a NASA project is developed in-house, it is common to find embedded Open Source Software within the code. It is often more efficient for a software engineer to use widely available and well tested code developed in the software community for common functions than to "reinvent the wheel."
3.4.1 What is Open Source Software?
3.4.2 Planning ahead for the inclusion of Open Source Software
Whether Open Source Software is acquired or developed by NASA or a NASA contractor, a usage policy should be established up front to avoid any possible legal issues that may arise. This policy may be developed in conjunction with advice from the Software Release Authority (even if you do not plan to release the software) and/or your Center's IP Legal Counsel.
3.4.3 Releasing NASA code containing Open Source Software
When software is released to another NASA Center, NASA project or external organization, it is important to inform the receiving party of any licenses and restrictions under which the software is released. It is important to note that additional software required to "run" the released software is not part of the software release. For example a web application that runs under the Apache Web Server does not need to include the Apache Public License as part of the relevant licenses.
Software releases are also performed when software is submitted for Space Act Awards, such as the NASA Software of the Year Award. For more information on software releases one should contact the Software Release Authority at the NASA Center at which the software is being or was developed.
Caution: Open Source Software may itself contain other Open Source Software!
3.4.4 Identifying and using high pedigree Open Source Software in NASA code
Going back to the NPR 7150.2, requirement 2.3.1.e, which states: "The software component is verified and validated to the same level of confidence as would be required of the developed software component." To achieve this level of confidence, it is recommended that software developers use only Open Source Software (and COTS, GOTS MOTS, legacy as well) software that has a high pedigree, i.e. a gold standard. Such Open Source Software will typically have the following characteristics:
- There should be a strong software development model including defined processes for bug reporting, which includes identification, triage, and correction.
- Code modification: Review and approval to commit fixes and features to the source code.
- Testing: Thorough descriptions of test cases, test runs, and test configurations.
- Code review (as part of the code modification process).
- Documentation: Detailed documentation should be updated.
- Discussion lists for questions: This may include a wiki, mailing list, or live chat site.
- Leadership: Ensures that the community works in a coordinated fashion to define target functionality for each release and overall product focus.
- Usually, a high quality, established open source project will have a large number of developers. Usage of the project's product(s) will occur across multiple industries both nationally and internationally.
- Metrics (e.g., number of developers) can be used to evaluate the quality of an Open Source project via sites that track a large proportion of Open Source Software projects (e.g., http://www.ohloh.net/). Norms, such as what constitutes a large number of developers, change as the number of Open Source projects and developers grow.
- The project should provide a listing of all Open Source Software included, embedded, or modified within the piece of Open Source Software.
- Open Source Software may provide more opportunity to perform V&V of the software to the same level of confidence as if obtained through a "development" process. Often Open Source Software project will provide online access to detailed development and test artifacts (as described above), which may be difficult to obtain from COTS vendors.
3.4.5 Procurement of software by NASA – Open Source Provisions
- Determine the ramifications of incorporating such external Open Source Software during the acquisition planning process specified in NASA FAR Supplement Subpart 1807.1, Acquisition Plans; and
- Consult with the Center Patent or IP Counsel early in the planning process (see 220.127.116.11) as the license under which the Open Source Software was acquired may negatively impact NASA's intended use."
3.5 Embedded Software
Embedded software applications written by/for NASA are commonly used by NASA for engineering software solutions. Embedded software is software specific to a particular application as opposed to general purpose software running on a desktop. Embedded software usually runs on custom computer hardware ("avionics"), often on a single chip.
Care must be taken when using vendor-supplied board support packages (BSPs) and hardware-specific software (drivers) which are typically supplied with off-the-shelf avionics systems. BSPs and drivers act as the software layer between the avionics hardware and the embedded software applications written by/for NASA. Most CPU boards have BSPs provided by the board manufacturer, or third parties working with the board manufacturer. Driver software is provided for serial ports, USB ports, interrupt controllers, modems, printers, and many other hardware devices
BSPs and drivers are hardware dependent, often developed by third parties on hardware/software development tools which may not be accessible years later. Risk mitigation should include hardware-specific software, such as BSPs, software drivers, etc.
Many BSPs and drivers are provided by board manufacturers as binary code only, which could be an issue if the supplier is not available and BSP/driver errors are found. It is recommended that a project using BSPs/drivers maintain a configuration managed version of any BSPs with release dates and notes. Consult with avionics (hardware) engineers on the project to see what actions, if any, that may be taken to configuration manage the BSPs/drivers.
Consideration should also be given to how BSP/driver software updates are to be handled, if and when they are made available, and how it will become known to the project that updates are available?
Vendor reports and user forums should be monitored from the time hardware and associated software are purchased through a reasonable time after deployment. Developers should monitor suppliers or user forums for bugs, workarounds, security changes, and other modifications to software that, if unknown, could derail a NASA project. Consider the following snippet from a user forum:
"Manufacturer Pt. No." motherboard embedded complex electronics contains malware
A "Manufacturer" support forum identifies "manufacturer's product" motherboards which contain harmful code. The embedded complex electronics for server management on some motherboards may contain malicious code. There is no impact on either new servers or non-Windows based servers. No further information is available regarding the malware, malware mitigation, the serial number of motherboards affected, nor the source
4. Small Projects
This requirement applies to all projects regardless of size.
6. Lessons Learned
The following information comes from the NASA Study on Flight Software Complexity listed in the reference section of this document:
"In 2007, a relatively new organization in DoD (the Software Engineering and System Assurance Deputy Directorate) reported their findings on software issues based on approximately 40 program reviews in the preceding 2½ years (Baldwin 2007). They found several software systemic issues that were significant contributors to poor program execution." Among the seven listed were the following on COTS:
- "Immature architectures, COTS integration, interoperability."
"Later, in partnership with the NDIA, they identified the seven top software issues that follow, drawn from a perspective of acquisition and oversight." Among the seven listed were the following on COTS:
- "Inadequate attention is given to total life cycle issues for COTS/NDI impacts on life cycle cost and risk."
"In partnership with the NDIA, they made seven corresponding top software recommendations." Among the seven listed were the following on COTS:
The following information is from Commercial Item Acquisition: Considerations and Lessons Learned July 14, 2000, Office of the Secretary of Defense:
The NASA Lessons Learned database contains the following lessons learned related to the user of commercial, government, and legacy software:
- "Simulate how the product will work on various customer platforms.
- "Model the field operations.
- "Develop in stages with ongoing integration and testing."