- 1. Purpose
- 2. Planning
- 3. Solicitation, Selection, and Award
- 4. Technical Monitoring and Quality Assurance
- 5. Contract Administration
- 6. Product Acceptance and Control
- 7. Contract Closeout
- 8. Useful Practices, Activities and Templates
- 9. Resources
This topic discusses guidance for projects implementing those requirements in NASA Procedural Requirement (NPR) 7150.2, NASA Software Engineering Requirements that address software acquisition. This guidance is intended for all persons responsible for the software acquisition process, from the planning stages through contract closeout. Acquisition may involve procedures and regulations external to the software community, including variations by contract type; therefore, it is important to consult Center guidance and coordinate acquisition activities among the proper stakeholders, including, but not limited to, software engineering, procurement, finance, and contracts.
Approve procurement plan.
Software Lead Engineer
Prepare procurement plan; prepare statement of work (SOW) software requirements and software data requirements for the contract; monitor execution of contract; conduct trade studies, engineering analyses.
Conduct trade studies, engineering analyses.
Contracting Officer (CO)
Prepare acquisition approach, prepare solicitation, guide proposal evaluation, prepare contracts, prepare modifications to contracts.
Contracting Officer's Representative (COR)
Work with CO to plan acquisition approach, prepare SOW, evaluate proposals, determine the technical adequacy of proposed approach, monitor technical implementation.
Software Technical Authority
Before contract release, verify that the SOW includes the complete flowdown of the Agency and Center software requirements [recommended practice].
Before software acquisition can be carried out, a need must be identified for which a solution is required. During the planning stage, various options to address the identified need are evaluated. The following are possible options:
- Acquire off-the-shelf (OTS) product.
- Develop/perform service in-house (make/buy decision).
- Contract development/service.
- Use/enhance existing product/service (consider reuse of existing software components for applicability to project).
If the solution to the need will involve software, NPR 7150.2 applies, and the acquisition planning guidance below supports project success:
- Define the scope of the system of interest.
- Identify the goals and objectives for the software portion of the system.
- Identify key stakeholders.
- Perform “make or buy” market research/trade studies to determine if an
- Establish criteria (and a plan) for the studies:
- Technical requirements (functional, operational, performance):
- NPR 7150.2 classification.
- Constraints and limitations (cost, schedule, resources).
- Use past studies, known alternatives, existing make/buy criteria.
- Conduct studies:
- Assess potential products and technologies.
- Assess how well technical requirements are addressed.
- Assess estimated costs, including support.
- Assess safety criticality.
- Identify risks (delivery, safety, development practices used by supplier, supplier track record, etc.).
- Assess provider business stability, past performance, ability to meet maintenance requirements, etc.
- Assess commercial off-the-shelf/government off-the-shelf/military off-the-shelf products for potential use. (See SWE-027.)
- Identify in-house capabilities to meet the need:
- Assess availability of existing products which could meet the need or be modified to meet the need.
- Assess availability of qualified personnel for development or modification activities.
- Assess estimated costs (time, personnel, materials, etc.), including support.
- Use past projects as basis, where appropriate.
- Identify risks.
- Determine if solution will be custom made, an existing product, or a modified existing product.
- Expected classification of the software to be acquired.
- Availability of in-house staff and funding resources.
- Availability of the software product(s).
- Projected licensing and support costs.
- List of potential suppliers.
- Security considerations.
- Potential risks related to supplier’s viability and past performance.
- Estimate of in-house versus acquisition costs (including OTS solutions and any associated costs for requirements not met by the OTS solution).
- Comparison of cost estimates to available funding.
- Risk assessment.
- Assumptions, observations, rationale, determining factors.
- Significant issues, impacts of each option.
- If solution is in-house development/service, an acquisition is no longer required.
- If solution is to acquire product/service, continue with this guidance as needed based on development under contract or purchase OTS solution.
- Other planning decisions resulting in best overall value to NASA.
- Description of chosen acquisition strategy.
based on requirements and “make or buy” decisions:
- Those directly concerned with, or affected by, the acquisition decision.
- May include management, the project team, procurement, customers, end users, and suppliers.
3. Solicitation, Selection, and Award
Once the planning activities for software acquisition have been completed and the decision has been made to acquire the software or software development services, a selection process needs to be followed to choose the best provider for the project. This process typically begins with development of a Statement of Work (SOW).
Typically, solicitations are prepared by the procurement or contracts department with input from project management and engineering. Solicitations are as complete as possible to ensure potential providers are aware of all tasks and activities for which they will be held responsible.
Checklists (e.g., the NASA Process Asset Library (PAL) contains checklists for NPR 7150.2 requirements by class and safety criticality) and solicitations for similar projects help ensure that all required activities are included in the solicitation. Example solicitations,
s, and work breakdown structures (
s) that are considered "best practices" are also helpful starting points; using problematic examples will only carry forward the problems exhibited or caused by those documents.
The following is a list of useful practices when documenting tasks and activities in the solicitation:
- Keep the task descriptions clear, concise, and in terms that providers will understand.
- Avoid over-specifying (specifying every item to the smallest detail leaving no options).
- Avoid under-specifying (providing insufficient or incomplete details).
- Clearly mark mandatory versus optional/recommended tasks, activities, standards, etc.
The following recommendations should be considered as part of the SOW development process. Additionally, a SOW checklist reference is included in the Useful Practices, Activities and Templates section (tab 8) of this guidance document.
- Develop solicitation, including
- Acceptance criteria (SWE-034SWE-034 - Acceptance Criteria).
- Solicitation constraints.
- Proper requirements from the software development perspective*:
- Software classification (from NPR 7150.2) and safety criticality (from Software Safety Litmus Test).
- Technical requirements.
- Adherence to requirements for safety-critical software (see SWE-134SWE-134 - Safety Critical Software Requirements, SWE-023SWE-023 - Software Safety, and NPR 7150.2, Appendix C).
- Development standard to be followed, if any.
- Development life cycle to be followed or indication that developer can choose appropriate life cycle (SWE-019SWE-019 - Software Life Cycle).
- Surveillance activities (and acquirer involvement), including monitoring activities, reviews, audits (SWE-045SWE-045 - Project Participation in Audits), decision points, meetings, etc.(SWE-039SWE-039 - Software Supplier Insight)
- Management and support requirements (project management, schedule and schedule updates (SWE-046SWE-046 - Supplier Software Schedule), configuration management, non-conformance and change tracking (SWE-043SWE-043 - Track Change Request), risk management, metrics collection, Independent Verification and Validation (IV&V) support, required records, traceability records (SWE-047SWE-047 - Traceability Data), electronic records and code access (SWE-042SWE-042 - Source Code Electronic Access), V&V, etc.)
- Requirements for maintenance, support, updates, new versions, training to be included in life-cycle and cost estimates.
- Concise task and deliverable descriptions, including delivery format (SWE-040SWE-040 - Access to Software Products).
- Media format for code deliverables (SWE-040SWE-040 - Access to Software Products).
- Templates or Data Item Descriptions (DIDs) for documentation deliverables.
- Complete set of deliverables with delivery dates, review periods, and acceptance procedures for each.
- Time period for responses to review findings, including making changes.
- Data Requirements Documents (DRDs) for deliverables, if appropriate.
- Government and contractor proprietary, usage, ownership, warranty, data, and licensing rights, including transfer.
- Requirement to include notice of use of open source software (SWE-041SWE-041 - Open Source Software Notification) in developed code.
- OTS software requirements (SWE-027SWE-027 - Use of Commercial, Government, and Legacy Software) (identify which requirements are met by
- List of all mandatory NASA software development standards and DIDs, as applicable.
- Requirements for non-expired
2. Ensure proper review of SOW before delivery to procurement/contracts official:
- Technical Authority to ensure proper flowdown of NPR 7150.2 requirements.
- Coordinate with the Safety and Mission Assurance Office to ensure all quality assurance requirements, clauses, and intended delegations are identified and included.
3. Identify potential suppliers.
4. Distribute solicitation package.
5. Evaluate proposals (typically an evaluation team), based on selection criteria established during the acquisition planning phase, including:
- Cost estimation comparisons.
- Evaluation of how well proposed solutions meet the requirements (including interface and technology requirements, NPR 7150.2 requirements).
- Staff available.
- Past performance.
- Software engineering and management capabilities.
- Prior expertise on similar projects.
- Available resources (facilities, hardware, software, training, etc.).
- Capability Maturity Model Integration (CMMI®) ratings.
- Other factors relevant to the project.
7. Negotiate, finalize and document contract:
- Based on SOW.
- Management reviews and meetings, such as:
- Formal reviews, such as those found in NPR 7123.1, NASA Systems Engineering Processes and Requirements, and NPR 7120.4, NASA Engineering and Program/Project Management Policy.
- Technical reviews.
- Progress reviews.
- Peer reviews (see Topic 7.10 - Peer Review and Inspections Including Checklists7.10 - Peer Review and Inspections Including Checklists in Handbook).
- Software quality assurance meetings.
- System integration test and verification meetings.
- System safety meetings.
- Configuration management meetings.
- Other relevant reviews for this project.
- Consider for inclusion in contract provisions (description of the method to be used) for verification of:
- Contractor handling of requirements changes.
- Accuracy of contractor transformation of high-level requirements into software requirements and detailed designs.
- Interface specifications between the contractor’s product and systems external to it.
- Adequacy of contractor’s risk management plan and its implementation in accordance with the required activities in the project Software Risk Management Plan.
- Adequacy of the contractor’s integration and test plan and its implementation in accordance with the required activities in the project Software Integration and Test Plan.
- Adequacy of the contractor’s configuration management plan and its implementation in accordance with the required activities in the project Software Configuration Management Plan.
- Consider for inclusion in the contract the content and frequency of progress reports and metrics submissions.
- Consider for inclusion in the contract identification of quality records to be maintained by the supplier.
- Consider for inclusion in the contract the delivery process and how it will be accomplished; if incremental development and delivery agreed upon, state how the validation process works, e.g., incremental validation, and whether it requires integration and test with software/hardware products developed by acquirer and/or other contractors or organizations (other institutes, universities, etc.).
- Consider for inclusion in the contract a policy for maintaining the software after delivery: Who is responsible for maintenance of the software, tools, testbeds, and documentation updates.
4. Technical Monitoring and Quality Assurance
Once the provider has been chosen, the acquisition process moves into a technical monitoring role. The following guidance should be included when establishing the process for provider monitoring and quality assurance:
- Provide technical requirements interpretation for contractor.
- Ensure contractor requirements documents meet original intent.
- Evaluate contractor progress with respect to cost.
- Periodically monitor contractor skill mix to ensure agreed-upon skills and experience levels are being provided.
- Oversee government-furnished equipment (GFE) to ensure equipment and information provided in timely manner.
- Periodically assess contractor processes to ensure conformance to process requirements stated in the contract.
- Review and assess adequacy of contractor-provided documentation and ensure contractor implementation of feedback, consider using Formal Inspections (SWE-087SWE-087 - Software Peer Reviews and Inspections for Requirements, Test Plans, Design, and Code) to accomplish this task.
- Track status considering the following example questions:
- Is the contractor meeting their staffing plan?
- Have the project and the contractor met the user’s needs?
- Does the contractor have stable, educated staff?
- Does the contractor’s project have adequate resources, e.g., adequate staffing and computer resources?
- Is there realistic planning/budgeting in place?
- Is the build plan being met?
- Does the contractor have a good understanding of what is required?
- Are the requirements stable?
- Is the completion of designed functionality visible?
- Is the evolving capability and performance of the contractor’s product likely to impact development on the acquirer side of the interface?
- Are integration and testing proceeding as planned?
- Is contractor cost/schedule performance on target?
- Is contractor developing a quality product?
9. Provide regular status reviews to higher-level management on contractor progress.
10. Regularly assess status of identified risks and provide reports during management reviews.
11. Software engineering should provide technical review to the level required to enhance the probability of mission success. (See the $paramlinktext of this topic for a list of areas to consider for software engineering technical review.)
5. Contract Administration
In addition to monitoring the selection provider’s progress and quality, contract administration activities are also carried out for the project. The following guidance should be included when establishing the process for contract administration:
- Regularly assess contractor financial data and invoices against budget.
- Work with Contracting Officer to ensure timely resolution of any contract-related issues.
- Work with Contracting Officer to ensure timely address of needed modifications to contract terms and conditions, as needed. These are primarily those affecting schedule, costs, services/products, resources (people, facilities), deliverables.
- Periodically evaluate contractor performance in manner consistent with contract and provide documented evaluation to Contracting Officer.
6. Product Acceptance and Control
Once the provider is ready to deliver the software product, the acquirer should have a process in place for review and acceptance of the product. The following guidance should be included when establishing the process for product acceptance:
- Review deliverables based on agreed-upon acceptance criteria (or generally accepted standards, if specific criteria have not been established), document results, and work with contractor to resolve acceptance issues.
- Typically, an acceptance test plan is created addressing the following:
- Acquirer and contractor roles and responsibilities.
- Defined test strategy.
- Defined test objectives.
- Defined acceptance criteria.
- Developed test Scenarios.
- Developed test scripts.
- Developed test matrix.
- Time and resources estimate.
- Approval cycle.
- Strategy for post-delivery problem resolutions.
- Once approved, the test plan is executed and results are documented:
- Select test tools.
- Select and train team members.
- Execute the test plan (manual and automated methods).
- Track test progress.
- Regression test.
- Document test results.
- Resolve problems.
- Typically, an acceptance test plan is created addressing the following:
2. Complete a Physical Configuration Audit (PCA) and a Functional Configuration Audit (FCA) to ensure that traceability is complete, that necessary waivers are in place, and that all required documentation has been developed.
3. Place formal deliverables under configuration control.
4. After acceptance of delivered products, support transition to an operational and/or maintenance environment.
7. Contract Closeout
The final acquisition step is to close out the contract. The following guidance should be included when establishing the process for contract close-out:
- Verify satisfaction of all contract terms and conditions, considering the following sample questions:
- Has the contract period of performance expired (level of effort type contract)?
- Have all deliverables been delivered (completion type contract)?
- Have all Contract Data Requirements List (CDRL) items been delivered and accepted?
- Was the contractor’s performance of the SOW acceptable?
- If the contract involved patent rights, has the final patent report been filed?
- Has the final invoice been received?
2. Verify return of all
, as appropriate.
3. Complete final reports as requested by Contracting Officer.
4. Provide final contractor performance evaluation to Contracting Officer.
5. Capture Lessons Learned, if not captured earlier in the project life cycle.