- 1. The Requirement
- 2. Rationale
- 3. Guidance
- 4. Small Projects
- 5. Resources
- 6. Lessons Learned
- 7. Software Assurance
4.6.4 The project manager shall complete, prior to delivery, verification that all software requirements identified for this delivery have been met, that all approved changes have been implemented, and that all defects designated for resolution prior to delivery have been resolved.
NPR 7150.2, NASA Software Engineering Requirements, does not include any notes for this requirement.
Click here to view the history of this requirement: SWE-194 History
1.3 Applicability Across Classes
Key: - Applicable | - Not Applicable
A & B = Always Safety Critical; C & D = Sometimes Safety Critical; E - F = Never Safety Critical.
Requirements are the basis for building a product that meets the needs of the project. They define the behavior of the system and the constraints under which the problem is to be solved. It is important that the implemented product is verified against the requirements to show that it actually meets all of the requirements.
However, requirements change over time and the documentation of the requirements is not always up to date. Also, for Agile approaches, not all of the requirements are necessarily documented upfront. It is critical that the Project Manager uses an appropriate method or tool to keep track of the requirements necessary for each delivery of the software and that these are kept up to date with any changes or defect resolutions.
The verification planning, execution, and results should include all requirements for that delivery. A traceability matrix that shows all of the planned and modified requirements to the test procedures/results is critical for showing compliance to this NPR requirement. Also critical are the test reports that show that each requirement verification passed.
This requirement applies to all NASA centers and Software Classes A, B, C, D, and F.
For the non-Agile lifecycle, an analysis of the baseline set of requirements and all change requests would need to be performed against the verification plans, procedures, and results to show that every requirement has been verified successfully.
For an Agile lifecycle, for each planned release, an analysis of the set of requirements associated with that release and all change requests associated with those requirements would need to be performed against the verification plans, procedures, and results to show that every requirement has been verified successfully. Additionally, any defect resolutions that were designated for inclusion in the release need to be analyzed against the verification procedures and results to ensure that they were resolved and successfully verified.
Keeping track of requirements changes can be done by several methods as described in SWE-053.
Verification is done by several methods. For guidance on inspections, see section 5.3 in the NPR. For guidance on testing, see section 4.5 in the NPR, specifically SWE-065, 066, 068, 071, and 191.
For this effort, having a complete bi-directionality between requirements, design, and verification is significant and will make the analysis easier.
4. Small Projects
No additional guidance is available for small projects.
No references have been currently identified for this SWE. If you wish to suggest a reference, please leave a comment below.
6. Lessons Learned
6.1 NASA Lessons Learned
No Lessons Learned have currently been identified for this requirement.
6.2 Other Lessons Learned
No other Lessons Learned have currently been identified for this requirement.
7. Software Assurance
7.1 Tasking for Software Assurance
Confirm that the project has identified the software requirements to be met, the approved changes to be implemented, and defects to be resolved for each delivery.
Confirm that the project has met all software requirements identified for the delivery.
Confirm that approved changes have been implemented and tested.
Confirm that the approved changes to be implemented and the defects to be resolved have been resolved.
Approve or sign-off on the projects delivered products.
7.2 Software Assurance Products
- Approvals/sign-offs on deliveries
- List of any risks and issues found with delivery.
Definition of objective evidence
- Evidence that delivery tasks 1 through 4 have been performed.
Objective evidence is an unbiased, documented fact showing that an activity was confirmed or performed by the software assurance/safety person(s). The evidence for confirmation of the activity can take any number of different forms, depending on the activity in the task. Examples are:
- Observations, findings, issues, risks found by the SA/safety person and may be expressed in an audit or checklist record, email, memo or entry into a tracking system (e.g. Risk Log).
- Meeting minutes with attendance lists or SA meeting notes or assessments of the activities and recorded in the project repository.
- Status report, email or memo containing statements that confirmation has been performed with date (a checklist of confirmations could be used to record when each confirmation has been done!).
- Signatures on SA reviewed or witnessed products or activities, or
- Status report, email or memo containing Short summary of information gained by performing the activity. Some examples of using a “short summary” as objective evidence of a confirmation are:
- To confirm that: “IV&V Program Execution exists”, the summary might be: IV&V Plan is in draft state. It is expected to be complete by (some date).
- To confirm that: “Traceability between software requirements and hazards with SW contributions exists”, the summary might be x% of the hazards with software contributions are traced to the requirements.
- # of software components (e.g. programs, modules, routines, functions, etc.) planned vs. # actually released in each build
- # of Software Requirements (e.g. Project, Application, Subsystem, System, etc.)
- # of software work product Non-Conformances identified by life-cycle phase over time
- # of software units planned vs. # actually built
- # of planned software requirements implemented in each build vs. # of actual software requirements implemented in each build
- # of detailed software requirements tested to date vs. total # of detailed software requirements
- # of tests successfully completed vs. total # of tests
- # of Requirements tested successfully vs. total # of Requirements
- # of Non-Conformances identified during each testing phase (Open, Closed, Severity)
For each delivery, review the delivery documentation and confirm that the document identifies the particular requirements that have been implemented for this delivery, any changes that are approved for implementation in this delivery and any defects that need to be resolved/implemented before delivery. It is usually helpful for the project to include a list of defects or discrepancies that have not been resolved in this delivery and will either need to be resolved in a future delivery or in maintenance. Delivery documentation for this information may include a delivery letter and a version description document.
The configuration management records for any changes, corrections or new capabilities/requirements should indicate which of the change requests have been approved and the planned delivery for their implementation. Confirm that all the requirements identified for this delivery have been successfully tested by reviewing the traceability matrices to see which tests map to the requirements for the delivery and checking the test reports for those requirements.
Review the set of approved changes for the software to be delivered and confirm that all of the approved changes that were scheduled to be implemented by delivery are actually in the software deliverable. Also, confirm that:
- The changes have been tested to confirm the approved change produces correct results,
- If the change was a defect, the changes fix the defect, and that checks have been made to catch any similar defects elsewhere in the deliverable software,
- Regression tests have been successfully run to ensure previously tested capabilities of the software still work. A full set of regression tests should be run on critical software.
For the set of defects identified in the software to be delivered, confirm that all defects planned for delivery have been resolved in one of the following ways:
- The defect is fixed and tested (confirming the above)
- The defect has an agreed-upon work-around (documented in the user documentation)
- The defect has been discussed with the customer and agreement was reached to defer the defect to a later build or maintenance or to leave the defect as is. If the agreement is to leave the defect “as is”, identify any risk associated with this decision
Software assurance personnel should sign off on the delivery documentation signifying that the delivery is complete and correct
For guidance on testing, see section 4.5 in the NPR, specifically SWE-065, 066, 068, 071, and 191. For more guidance on managing changes and release management, see SWE-053, 080 and 085.