1. Requirements

2.5.2 The project shall assess options for software acquisition versus development.

1.1 Notes

The assessment can include risk, cost, and benefits criteria for each of the options listed below:

• Acquire an off-the-shelf software product that satisfies the requirement.
• Develop the software product or obtain the software service internally.
• Develop the software product or obtain the software service through contract.
• Enhance an existing software product or service.

Risks are considered in software make/buy and acquisition decisions. The project needs to ensure that software products used in the design or support of human space flight components or systems include a level of rigor in risk mitigation as a software management requirement, regardless of software classification. The level of documentation needed for risk identification and tracking is defined by the Center processes.

1.2 Applicability Across Classes

2. Rationale

When making any decision, it is important to assess the options available in order to obtain the greatest value and benefit. Software development is no different. Choices need to be assessed in order to identify the best use of available resources (budget, time, personnel, etc.) to address a defined and scoped need while providing the greatest benefit with the least risk to the project.

3. Guidance

When assessing solutions for software acquisition versus development, there are four possible options:

• Acquire an off-the-shelf software product that satisfies the requirement.
• Develop the software product or obtain the software service internally.
• Develop the software product or obtain the software service through contract.
• Enhance an existing software product or service.

Each option has its own benefits, costs, and risks which should be identified through trade studies (for off-the-shelf products), internal assessments (for existing products), and cost-benefit analyses. Checklists of questions to ask when assessing acquisition versus development can be found in SWE-027 of this handbook.

The team should assess existing software products, whether off-the-shelf or in-house, to identify how well they meet the need of the current project and whether they are suitable for the intended environment. The following information should be weighed against the defined need, architecture, environment, requirements, safety classification, budget, etc. of the current project:

• Features/functionality/capabilities.
• Documentation.
• Test results.
• Performance record.
• Safety record.
• Licensing, maintenance, integration, and support costs.
• Any other relevant information.

The project responsible for procuring off-the-shelf software is responsible for documenting, prior to procurement, a plan for verifying and validating the off-the-shelf software to the same level of confidence that would be needed for an equivalent class of software if obtained through a "development" process. For more detail, see SWE-027 - Use of Commercial, Government, and Legacy Software.

For development, whether internal or external, consider the following information:

• Personnel skill sets, experience, availability.
• Cost associated with training, tools, post-development maintenance and support.
• Company reputation, track record, history, etc. (for contracted development).
• Overall life-cycle cost, including cost of integration with existing software components.
• Intellectual Property rights.
• Cost and availability of workforce should follow-on work be required.
• Insight into development processes.
• Schedule associated with procurement (sole source, competitive, task order, etc.) for procured software or a contracted development.
• Life-cycle supportability risks.
• Complexity of effort and extent of modifications required.

Identify risks associated with each assessed option, including:

• Technical risks.
• Supplier risks, including track record and support risks.
• Cost and schedule risks.

Involve the relevant stakeholders in the assessment process to benefit from their experience and ensure all key information is considered. Consider the following, as applicable:

• Technical personnel.
• Management.
• Contracts.
• Procurement.
• End users.
• Customers.
• Technical Authority.

See the Topic 7.03 - Acquisition Guidance topic in this Handbook for additional guidance. The references in this topic may also provide additional guidance on assessing acquisition versus development options.

Additionally, Center procedures addressing decision analysis and resolution may be helpful in planning and carrying out the assessment and selection process.

4. Small Projects

While assessing all available options is important for any software development project, it may be even more important for projects with limited budgets, personnel, or both. Small projects need to evaluate their available resources against the possible solutions to find the best fit with the least risk.

Use of existing trade studies and market analyses may reduce the cost and time of assessing available options.

5. Resources

6. Lessons Learned

A documented lesson from the NASA Lessons Learned database notes the following topics that should be kept in mind when assessing software acquisitions versus development. While many of these lessons seem hardware-oriented, some of these lessons can also be applied to software ⁵⁵¹:

• Lessons Learned From Flights of "Off the Shelf" Aviation Navigation Units on the Space Shuttle, GPS (Talk to those that have used the product before.) Lesson Number 1370: "Outside consultants, who do not have a stake in the choice of a particular unit, should be used. Such consultants have "hands on experience" ... and can be an important information source concerning their design, integration and use. Consultants who have participated in previous integrations will have knowledge of problems that other users have encountered. Consultants and other users can also provide valuable insight into the rationale and requirements that governed the original design of the unit. This information is invaluable ... for identifying technical, cost and schedule risks associated with a particular ... unit ...."

• Lessons Learned From Flights of "Off the Shelf" Aviation Navigation Units on the Space Shuttle, GPS ("Plug And Play" versus development.) Lesson Number 1370: "The fact that a unit is in mass production and is a proven product does not mean that its integration to a different vehicle will be a simple, problem free 'plug and play' project. A difference in application (such as aviation versus space flight) will result in the manifestation of firmware issues that may not have appeared in the original application. Unique data interfaces used by manned and some unmanned spacecraft avionics may require modification of the unit. Power supply changes and radiation hardening may also have to be performed." While this lesson describes hardware acquisitions, software acquisitions should also keep this lesson in mind because projects have differences that can affect the suitability of software for a particular application.

• Lessons Learned From Flights of "Off the Shelf" Aviation Navigation Units on the Space Shuttle, GPS (Pay attention to "Technical Risk.") Lesson Number 1370:  "Project management may focus mainly on risk to cost and schedule, with little attention paid to technical risk. GPS project management kept Shuttle Program management well aware of the nature of a 'success oriented' approach and that cost and schedule could be impacted. Analysis at the start of a project should be conducted to determine risk to cost and schedule based on the technology level, the maturity of the technology and the difference between the planned application and the application for which the box was designed originally. Software complexity should also be examined. Failure to account for technical risk can lead to cost and schedule problems."

• * Lessons Learned From Flights of "Off the Shelf" Aviation Navigation Units on the Space Shuttle, GPS (Provide guidelines for COTS and "Faster-Better-Cheaper" implementation). Lesson Number 1370:* "A key lesson from unmanned spacecraft failures and DoD software programs is that one must understand how to properly use COTS (Commercial Off the Shelf) products and apply 'faster-better-cheaper' principles."

    "A COTS policy is needed to help prevent cost, schedule and technical difficulties from imperiling projects that use COTS. Criteria for determining whether a COTS approach can be taken must be determined. Of prime importance is defining the level of insight needed into vendor software, software maintenance and certification processes."

    "Problems in COTS projects can arise when requirements are levied on the product that the vendor did not originally intend for the unit to meet. Using COTS may mean either compromising requirements on the COTS unit or on the integrated system. Whether or not new requirements have to be applied to the unit is a critical decision. Unfortunately, new requirements may not be recognized until the COTS product experiences difficulties in the testing and integration phases of the project."

    "The Shuttle Program created COTS/MOTS (Modified Off the Shelf) software guidelines for varying levels of application criticality. This recommended policy defines what considerations should be made before deciding to procure a COTS/MOTS product. The following should be examined based on the criticality (impact of failure on safety of flight or mission success) of the application and product in question:

• "Certification Plan – How much of the vendors in-house certification can be relied upon? For critical applications, additional testing will be needed if access to test results, source code and requirements documents is not allowed. Can the unit be certified to a level commensurate with the criticality of the application?
• "Vendor Support – This should cover the certification process and the system life cycle. The level of support should be defined based on the criticality of the system.
• "Product Reliability – Vendor development and certification processes for both hardware and software should be examined.
• "Trade Studies – Define 'must meet,' 'highly desirable' and 'nice to have' requirements. Ability of the unit to meet those requirements, and at what cost, will be a major deciding factor in the COTS decision. Identify loss of operational and upgrade flexibility as well technical risks and cost associated with the product. Examine the impact of the product on the integrated system, including hardware and software interface changes. Compare the proposed COTS products to a custom developed product. Assess life expectancy of the product and its track record in the market place.
• "Risk Mitigation – Identify areas that increase risk, such as lack of support if the vendor goes out of business or the product is no longer produced. Ensuring vendor support over the product life cycle can mitigate risk, along with gaining access to source code, design requirements, verification plans and test results. Off-line simulations of the product should also be considered. Can access be obtained to vendor information on product issues discovered by other users?

"Trade studies and risk identification must be performed before committing to the use of a particular unit and integration architecture."