- 1. The Requirement
- 2. Rationale
- 3. Guidance
- 4. Small Projects
- 5. Resources
- 6. Lessons Learned
- 7. Software Assurance
4.6.3 The project manager shall complete and deliver the software product to the customer with appropriate records, including as-built records, to support the operations and maintenance phase of the software’s life cycle.
NPR 7150.2, NASA Software Engineering Requirements, does not include any notes for this requirement.
Click here to view the history of this requirement: SWE-077 History
1.3 Applicability Across Classes
Key: - Applicable | - Not Applicable
A & B = Always Safety Critical; C & D = Sometimes Safety Critical; E - F = Never Safety Critical.
The ultimate goal of software development is to provide a product to the customer. It is necessary that documentation accompany that delivery to ensure proper understanding, use and maintenance of the delivered product.
Typical contents of a software delivery package for a completed software project include the source files, executables (the product), a user's manual, and a version description document that describes the delivered product and contains procedures for creating executable software and modifying the software. Suggestions for documentation contents are defined in 7.18 - Documentation Guidanceof this Handbook. Open-source software licenses are also considered part of the delivery package and are reviewed by the Center's Chief of Patent/Intellectual Property Counsel before being accepted. Other documentation considered for delivery as appropriate for the project and its software classification (SWE-020) includes:
- Summary and status of all accepted Change Requests to the baselined Software Requirements Specifications.
- Summary and status of all major software capability change since baselining of the Software Design Documents.
- Summary and status of all major software tests (including development, verification, and performance testing).
- Summary and status of all Problem Report written against the software.
- Summary and status of all software requirements deviations and waivers.
- Summary and status of all software user notes.
- Summary and status of all quality measures historically and for this software.
Definition of openwork, if any.
- Software configuration records defining the verified and validated software, including requirements verification data (e.g., requirements verification matrix).
- The final version of the software documentation, including the final Software Version Description document(s).
- Summary and status of any open software-related risks.
In addition to the items listed in this requirement, consider the following items for the software delivery package:
- Certifications. 278
- Software safety plan. 271
- Hazard analyses. 271
- Safety-critical software development audit reports. 271
- Safety-related verification reports. 271
- Installation instructions, including a description of the hardware environment. 276
- Operational constraints, including environmental limitations. 276
- Configuration files. 276
- Project documentation (e.g., software development/management plan (SDP/SMP), assurance plan, software requirements specification (SRS), design documents, configuration management plan (CMP), test plan). 276
- Testing performed and the results of those tests. 276
- Training manuals 041, training agreements. 224
- Maintenance/service/support agreement, processes, procedures. 224
- Development environment (any specialized hardware and software needed to build the executable software during the maintenance phase).
- Hardware needed to test the software if specialized hardware is needed (for maintenance).
Once the project team establishes the set of documentation to be delivered with the software, the team keeps documents up-to-date throughout the project life cycle to avoid any delays in delivery. Near the end of the project life cycle, the team updates any inconsistent documentation to ensure it includes the information needed to maintain the software in the future.
Before software delivery, a check of the software is made to verify that it conforms as-built to its technical documentation. When mismatches occur this close to software delivery, the project team updates the documentation to match the delivered software prior to delivery to the customer. It is important to perform an audit prior to delivery to ensure that "all delivered products are complete, contain the proper versions, and that all discrepancies, openwork, and deviations and waivers are properly documented and approved." 278
Both the delivered software and the delivered documentation are generated/pulled from the project's configuration management system (SWE-085) as a baseline to ensure the latest versions are delivered to the customer.
Delivery package contents from a contracted software provider need to be fully described in the contract to ensure the acquirer receives all critical information required to operate and maintain the software. Other contract considerations related to delivery include:
- Ownership and delivery of source code.
- Usage considerations for off-the-shelf (OTS) software.
- Delivery format, security. 273 and method for all deliverables.
- Acceptance criteria (SWE-034).
See topic 7.3 - Acquisition Guidance in this Handbook for additional guidance regarding delivery for contracted software development.
NASA users should consult Center Process Asset Libraries (PALs) for Center-specific guidance related to the delivery of software products.
NASA-specific software delivery information and resources are available in Software Processes Across NASA (SPAN), accessible to NASA users from the SPAN tab in this Handbook.
Additionally, guidance related to the delivery of software products may be found in the following related requirements in this Handbook:
4. Small Projects
No additional guidance is available for small projects.
- CMMI Development Team (2010). "CMMI for Development, Version 1.3: Improving processes for developing better products and services,"CMMI Development Team (2010). CMU/SEI-2010-TR-033, Software Engineering Institute.
6. Lessons Learned
6.1 NASA Lessons Learned
A documented lesson from the NASA Lessons Learned database notes the following:
- International Space Station (ISS) Program/Command and Data Handling/Firmware Documentation (Firmware Documentation.) Lesson Number 1024 534: "NASA should ensure that all firmware code...is properly documented and archived for future reference. Further, NASA should ensure that it retains the rights to such software. Direction to deliver copies of the documentation (requirement, design, test, etc.) of the firmware controller software prepared as part of their software development process is being given to each vendor."
6.2 Other Lessons Learned
No other Lessons Learned have currently been identified for this requirement.
7. Software Assurance
7.1 Tasking for Software Assurance
Confirm that the correct version of the products is provided, including as-built documentation and project records.
Perform audits on the configuration management processes to verify that all products are being delivered and are the correct versions.
7.2 Software Assurance Products
- Software Configuration Management Baseline and Process/Procedure Audit Report
- Confirmation that Task 1 has occurred.
- Configuration management audits with audit findings to verify delivery products.
- Configuration management process audits with findings, issues identified.
Definition of objective evidence
- Evidence confirming that the correct version of the products is provided, including as-built documentation and project records.
Objective evidence is an unbiased, documented fact showing that an activity was confirmed or performed by the software assurance/safety person(s). The evidence for confirmation of the activity can take any number of different forms, depending on the activity in the task. Examples are:
- Observations, findings, issues, risks found by the SA/safety person and may be expressed in an audit or checklist record, email, memo or entry into a tracking system (e.g. Risk Log).
- Meeting minutes with attendance lists or SA meeting notes or assessments of the activities and recorded in the project repository.
- Status report, email or memo containing statements that confirmation has been performed with date (a checklist of confirmations could be used to record when each confirmation has been done!).
- Signatures on SA reviewed or witnessed products or activities, or
- Status report, email or memo containing Short summary of information gained by performing the activity. Some examples of using a “short summary” as objective evidence of a confirmation are:
- To confirm that: “IV&V Program Execution exists”, the summary might be: IV&V Plan is in draft state. It is expected to be complete by (some date).
- To confirm that: “Traceability between software requirements and hazards with SW contributions exists”, the summary might be x% of the hazards with software contributions are traced to the requirements.
- # of Configuration Management Audits conducted by the project – Planned vs. Actual
- # of Non-Conformances per audit (including findings from process and compliance audits, process maturity)
- Trends of # of Non-Conformances from audits over time (Include counts from process and standards audits and work product audits.)
- # of Open vs. Closed Audit Non-Conformances over time
- # of Compliance Audits planned vs. # of Compliance Audits performed
- # of software process Non-Conformances by life-cycle phase over time
- # of Non-Conformances identified in release documentation (Open, Closed)
- # of software components (e.g. programs, modules, routines, functions, etc.) planned vs. # actually released in each build
Software assurance will confirm that:
- all the software planned for the delivery has been completed and
- all software has been tested and has successfully met the acceptance test criteria, verifying that all the capabilities as per the requirements are ready for operations and maintenance.
- the software includes all the approved changes and the changes have been tested,
- all defects approved for implementation are implemented and have been successfully tested (Where similar software exists elsewhere in the system being delivered, confirm that developers have checked to see that the defect does not exist in those places)
- all planned documentation for the build is completed and included with the delivery (user manuals, as-built documentation, operations manuals, build procedures or scripts, regression test sets with expected results, maintenance handbooks, etc.)
- delivery includes a list of any changes or defects that have not been implemented and their status (deferred to maintenance, accepted with a workaround, accepted as is)
- all other products needed for operations and maintenance are included in the delivery
To confirm that all the software has been successfully tested and meets the acceptance criteria, software assurance should perform or participate in a functional configuration audit (FCA). This confirms that all requirements for the build have been tested and have successfully passed. Software assurance should confirm that any changes or defects implemented in the software have also been successfully tested, including regression testing.
In some cases, the software system may be delivered with known defects or non-functional capabilities if those are documented in the delivery documentation and the customer has agreed to accept the system with those defects. If the system is being delivered with known defects, the software assurance personnel should highlight any risks that those defects cause in the system, particularly if there are risks to the safety, security or reliability of the software system. If risks are severe enough in those areas, software assurance might consider recommending not delivering the system until those risks are addressed.
In order the confirm that all products are included in the delivery in the correct versions, including the as-built documentation and project records, a physical configuration audit (PCA) against the documentation defining the delivery items needs to be performed. Software assurance will either perform this audit or participate in the audit and will sign off that the delivery is complete (i.e., it contains all the items it is supposed to have, in their correct versions for the delivery). The delivery items will be defined in the project documentation defining the planned builds and deliveries. Recorded version numbers of the items for delivery will be in the project configuration management system recording the baseline version planned for the delivery. These versions are usually also listed in the delivery letter or version description document (see VDD - Version Description Document) which are part of the delivery package. Other items that might be necessary for the delivery package are mentioned in the software guidance section of the requirement.
These tasks can be accomplished by Software Assurance reviewing the results of the configuration audits. Every task that involves performing an audit should also clarify that all audit findings are promptly shared with the project will be addressed in the handbook guidance.