1. Requirements

1.1 Notes

The assessment can include risk, cost, and benefits criteria for each of the options listed below:

a. Acquire an off-the-shelf software product that satisfies the requirement.

b. Develop a software product or obtain the software service internally.

c. Develop the software product or obtain the software service through contract.

d. Enhance an existing software product or service.

e. Reuse an existing software product or service.

f. Source code available external to NASA.

See the NASA Software Engineering Handbook for additional detail.

1.2 History

1.3 Applicability Across Classes

Key:    - Applicable | - Not Applicable

2. Rationale

When making any decision, it is important to assess the options available to obtain the greatest value and benefit.  Software development is no different.  Choices need to be assessed to identify the best use of available resources (budget, time, personnel, etc.) to address a defined and scoped need while providing the greatest benefit with the least risk to the project.

3. Guidance

3.1 Acquisition Versus Development Options

When assessing solutions for software acquisition versus development, there are five possible options:

• Acquire an off-the-shelf software product that satisfies the requirement.
• Develop a software product or obtain a software service internally.
• Develop the software product or obtain the software service through the contract.
• Enhance an existing software product or service.
• Reuse an existing software product or service.

Each option has its benefits, costs, and risks which should be identified through trade studies (for off-the-shelf products), internal assessments (for existing products), and cost-benefit analyses. Checklists of questions to ask when assessing acquisition versus development can be found in SWE-027 - Use of Commercial, Government, and Legacy Software in this handbook.

3.2 Risks in Make Versus Buy Decisions

Risks are considered in software to make/buy and acquisition decisions.  The project needs to ensure that software products used in the design or support of human space flight components or systems include a level of rigor in risk mitigation as a software management requirement, regardless of software classification.  The level of documentation needed for risk identification and tracking is defined by the Center processes.

3.3 Reuse of Existing Software Products

The team should assess existing software products, whether off-the-shelf or in-house, to identify how well they meet the need of the current project and whether they are suitable for the intended environment. The following information should be weighed against the defined need, architecture, environment, requirements, safety classification, budget, etc. of the current project:

• Features/functionality/capabilities.
• Documentation.
• Test results.
• Performance record.
• Safety record.
• Licensing, maintenance, integration, and support costs.
• Any other relevant information.

The project responsible for procuring off-the-shelf software is responsible for documenting, prior to procurement, a plan for verifying and validating the off-the-shelf software to the same level of confidence that would be needed for an equivalent class of software if obtained through a "development" process. For more detail, see SWE-027 - Use of Commercial, Government, and Legacy Software.

See also 6.3 - Checklist for Choosing a Real Time Operating System (RTOS), PAT-025 - Checklist for Choosing a Real Time Operating System (RTOS), 

3.4 Developing Software

For development, whether internal or external, consider the following information:

• Personnel skill sets, experience, availability.
• The cost is associated with training, tools, post-development maintenance, and support.
• Company reputation, track record, history, etc. (for contracted development).
• Overall life cycle cost, including the cost of integration with existing software components.
• Intellectual property rights.
• Cost and availability of the workforce should follow-on work be required.
• Insight into development processes.
• Schedule associated with procurement (sole source, competitive, task order, etc.) for procured software or a contracted development.
• Life cycle supportability risks.
• The complexity of effort and extent of modifications required.

3.5 Making the Decision

Identify risks associated with each assessed option, including:

• Technical risks.
• Supplier risks, including track record and support risks.
• Cost and schedule risks.

Involve the right stakeholders in the assessment process to benefit from their experience and consider all key information.  Consider the following, as applicable:

• Technical personnel.
• Management.
• Contracts.
• Procurement.
• End users.
• Customers.
• Technical Authority.

See 7.03 - Acquisition Guidance in this Handbook for additional guidance on this topic. The references in this topic may also provide additional guidance on assessing acquisition versus development options. See also SWE-015 - Cost Estimation, SWE-151 - Cost Estimate Conditions

Additionally, Center procedures addressing decision analysis and resolution may be helpful in planning and carrying out the assessment and selection process.

The Agency Software Manager can be used as a resource for this confirmation.

3.6 Additional Guidance

Additional guidance related to this requirement may be found in the following materials in this Handbook:

3.7 Center Process Asset Libraries

See the following link(s) in SPAN for process assets from contributing Centers (NASA Only). 

4. Small Projects

While assessing all available options is important for any software development project, it may be even more important for projects with limited budgets, personnel, or both. Small projects need to evaluate their available resources against the possible solutions to find the best fit with the least risk.

The use of existing trade studies and market analyses may reduce the cost and time of assessing available options.

5. Resources

5.1 References

5.2 Tools

6. Lessons Learned

6.1 Lessons Learned

A documented lesson from the NASA Lessons Learned database notes the following topics that should be kept in mind when assessing software acquisitions versus development. While many of these lessons seem hardware-oriented, some of these lessons can also be applied to software ⁵⁵¹:

• Lessons Learned From Flights of "Off the Shelf" Aviation Navigation Units on the Space Shuttle, GPS (Talk to those that have used the product before.) Lesson Number 1370: "Outside consultants, who do not have a stake in the choice of a particular unit, should be used. Such consultants have "hands-on experience" ... and can be an important information source concerning their design, integration, and use. Consultants who have participated in previous integrations will know problems that other users have encountered. Consultants and other users can also provide valuable insight into the rationale and requirements that governed the original design of the unit. This information is invaluable ... for identifying technical, cost, and schedule risks associated with a particular ... unit ...."

• Lessons Learned From Flights of "Off the Shelf" Aviation Navigation Units on the Space Shuttle, GPS ("Plug And Play" versus development.) Lesson Number 1370: "The fact that a unit is in mass production and is a proven product does not mean that its integration to a different vehicle will be a simple, problem-free 'plug and play' project. A difference in the application (such as aviation versus space flight) will result in the manifestation of firmware issues that may not have appeared in the original application. Unique data interfaces used by manned and some unmanned spacecraft avionics may require modification of the unit. Power supply changes and radiation hardening may also have to be performed." While this lesson describes hardware acquisitions, software acquisitions should also keep this lesson in mind because projects have differences that can affect the suitability of the software for a particular application.

• Lessons Learned From Flights of "Off the Shelf" Aviation Navigation Units on the Space Shuttle, GPS (Pay attention to "Technical Risk.") Lesson Number 1370:  "Project management may focus mainly on risk to cost and schedule, with little attention paid to technical risk. GPS project management kept Shuttle Program management well aware of the nature of a 'success-oriented' approach and that cost and schedule could be impacted. Analysis at the start of a project should be conducted to determine the risk to cost and schedule based on the technology level, the maturity of the technology, and the difference between the planned application and the application for which the box was designed originally. Software complexity should also be examined. Failure to account for technical risk can lead to cost and schedule problems."

• Lessons Learned From Flights of "Off the Shelf" Aviation Navigation Units on the Space Shuttle, GPS (Provide guidelines for COTS and "Faster-Better-Cheaper" implementation).Lesson Number 1370: "A key lesson from unmanned spacecraft failures and DoD software programs is that one must understand how to properly use commercial off the shelf (COTS) products and apply 'faster-better-cheaper' principles."

"A COTS policy is needed to help prevent cost, schedule, and technical difficulties from imperiling projects that use COTS. Criteria for determining whether a COTS approach can be taken must be determined. Of prime importance is defining the level of insight needed into vendor software, software maintenance, and certification processes."

"Problems in COTS projects can arise when requirements are levied on the product that the vendor did not originally intend for the unit to meet. Using COTS may mean either compromising requirements on the COTS unit or the integrated system. Whether or not new requirements have to be applied to the unit is a critical decision. Unfortunately, new requirements may not be recognized until the COTS product experiences difficulties in the testing and integration phases of the project."

"The Shuttle Program created COTS/MOTS (modified off the shelf) software guidelines for varying levels of application criticality. This recommended policy defines what considerations should be made before deciding to procure a COTS/MOTS product. The following should be examined based on the criticality (impact of failure on the safety of flight or mission success) of the application and product in question:

• "Certification Plan – How much of the vendors' in-house certification can be relied upon? For critical applications, additional testing will be needed if access to test results, source code, and requirements documents are not allowed. Can the unit be certified to a level commensurate with the criticality of the application?
• "Vendor Support – This should cover the certification process and the system life cycle. The level of support should be defined based on the criticality of the system.
• "Product Reliability – Vendor development and certification processes for both hardware and software should be examined.
• "Trade Studies – Define 'must meet,' 'highly desirable' and 'nice to have' requirements. The ability of the unit to meet those requirements, and at what cost, will be a major deciding factor in the COTS decision. Identify loss of operational and upgrade flexibility as well as technical risks and cost associated with the product. Examine the impact of the product on the integrated system, including hardware and software interface changes. Compare the proposed COTS products to a custom-developed product. Assess the life expectancy of the product and its track record in the marketplace.
• "Risk Mitigation – Identify areas that increase risk, such as lack of support if the vendor goes out of business or the product is no longer produced. Ensuring vendor support over the product life cycle can mitigate risk, along with gaining access to source code, design requirements, verification plans, and test results. Off-line simulations of the product should also be considered. Can access be obtained to vendor information on product issues discovered by other users?

"Trade studies and risk identification must be performed before committing to the use of a particular unit and integration architecture."

6.2 Other Lessons Learned

No other Lessons Learned have currently been identified for this requirement.

7. Software Assurance

7.1 Tasking for Software Assurance

7.2 Products

• Assessment of Risks in Acquisition vs Development Decisions
• Confirm that the options for software acquisition versus development have been evaluated.
• Confirm the flow down of applicable software engineering, software assurance, and software safety requirements on all acquisition activities. (NPR 7150.2 and NASA-STD-8739.8).

• Objective evidence that the confirmations for Tasks 1 and 2 (above) have occurred. (Possible evidence might be: review comments on trade study, risks, and concerns, comments on RFP/Contract requirements, etc.)

• Provide any risks and concerns to the project manager and engineering organization.

  
  

  Objective Evidence

  Statement Of Work (SOW) for each acquisition that includes software.  The SOW should have the flow down of applicable software engineering, software assurance, and software safety requirements (NPR 7150.2 and NASA-STD-8739.8) on all acquisition activities. 

   Definition of objective evidence

  Objective evidence is an unbiased, documented fact showing that an activity was confirmed or performed by the software assurance/safety person(s). The evidence for confirmation of the activity can take any number of different forms, depending on the activity in the task. Examples are:

  • Observations, findings, issues, risks found by the SA/safety person and may be expressed in an audit or checklist record, email, memo or entry into a tracking system (e.g. Risk Log).
  • Meeting minutes with attendance lists or SA meeting notes or assessments of the activities and recorded in the project repository.
  • Status report, email or memo containing statements that confirmation has been performed with date (a checklist of confirmations could be used to record when each confirmation has been done!).
  • Signatures on SA reviewed or witnessed products or activities, or
  • Status report, email or memo containing a short summary of information gained by performing the activity. Some examples of using a “short summary” as objective evidence of a confirmation are:
    • To confirm that: “IV&V Program Execution exists”, the summary might be: IV&V Plan is in draft state. It is expected to be complete by (some date).
    • To confirm that: “Traceability between software requirements and hazards with SW contributions exists”, the summary might be x% of the hazards with software contributions are traced to the requirements.
  • The specific products listed in the Introduction of 8.16 are also objective evidence as well as the examples listed above.

7.3 Metrics

• # of Software Requirements that do not trace to a parent requirement
• # of Risks by Severity (e.g., red, yellow, green) over time
• # of Risks with mitigation plans vs. total # of Risks
•  # of Risks trending up over time
•  # of Risks trending down over time

See also Topic 8.18 - SA Suggested Metrics

7.4 Guidance

Planning the software assurance on any project requires an understanding of the project’s function, needs, and risk posture as well as Software Class and criticality. Software Assurance needs to work with the software development team to assure that the software development processes are planned well and are based on the project and software criteria, as appropriate.

Software Assurance also needs to assure any acquisition process is adequate and complete and that criteria are set up to eventually assure the delivery of needed software products and reports.  Once these criteria are established, software assurance can create their plans and assure the appropriate SA needs are provided, e.g. sufficient personnel to perform the SA activities, any needed training on the project development processes and functionality, needed tools, and access to project data, products and activities.

When the software project is in the process of determining whether to “make or buy” their software, the software assurance should review the process used to make the decision. Although the process is often not followed with strict formality, these basic steps below should be considered:

• Use Guidelines for Decision Analysis
• Establish Evaluation Criteria
• Identify Alternative Solutions
• Select Evaluation Methods
• Evaluate Alternatives
• Select Solutions
• Document Solution and Rationale

Software assurance should evaluate the alternatives identified and investigated review the rationale used to make the selection. SA should document any risks identified with the choice. Some risk areas to consider include:

Will long-term maintenance support the products? Will the provider support requested changes to the product? Would update costs be reasonable? Does the product have a government-approved license? Is the source code available? Is the provider a stable company (that will be around for future support)? Is any new technology being proposed/used? Will there be any security risks?

Open Source licenses need to be reviewed by the Center Chief of Patent/Intellectual Property Counsel before being accepted into software development projects.

The Software assurance organization supports acquisition in determining whether all the appropriate SA support activities are included in the solicitations, contract, or task order, and in evaluating whether the proposed SA support activities have realistic cost estimates. The software assurance organization should also be supporting any acquisitions by making sure that all the requirements are flowed down to the contractor. These include at a minimum, NPR-7150.2, NASA-STD--8739.8, any Center level requirements, as well as the requirements for the project, including metrics and other deliverables needed for insight/oversight.

All solicitations should be checked to see if there is software included. Often, the software is included in the products such as tools, facilities, cameras, test equipment, instruments, etc. Failure to assure COTS and embedded software in purchased software has led to failures or serious losses.

Any risks or issues with decision or rationale are brought to the attention of management.

The Agency Software Manager can be used as a resource for this confirmation.

7.5 Additional Guidance

Additional guidance related to this requirement may be found in the following materials in this Handbook: