1. Requirements

4.6.4 The project manager shall complete, prior to delivery, verification that all software requirements identified for this delivery have been met or dispositioned, that all approved changes have been implemented, and that all defects designated for resolution prior to delivery have been resolved. 

1.1 Notes

NPR 7150.2, NASA Software Engineering Requirements, does not include any notes for this requirement.

1.2 History

1.3 Applicability Across Classes

Key:    - Applicable | - Not Applicable

2. Rationale

Requirements are the basis for building a product that meets the needs of the project. They define the behavior of the system and the constraints under which the problem is to be solved. The implemented product must be verified against the requirements to show that it meets all of the requirements.

However, requirements change over time and the documentation of the requirements is not always up to date. Also, for Agile approaches, not all of the requirements are necessarily documented upfront. It is critical that the Project Manager uses an appropriate method or tool to keep track of the requirements necessary for each delivery of the software and that these are kept up to date with any changes or defect resolutions.

The verification planning, execution, and results should include all requirements for that delivery. A traceability matrix that shows all of the planned and modified requirements for the test procedures/results is critical for showing compliance with this NPR requirement. Also critical are the test reports that show that each requirement verification passed.

3. Guidance

This requirement applies to all NASA centers and Software Classes A, B, C, D, and F.

For the non-Agile life cycle, an analysis of the baseline set of requirements and all change requests would need to be performed against the verification plans, procedures, and results to show that every requirement has been verified successfully.

For an Agile life cycle, for each planned release, an analysis of the set of requirements associated with that release and all change requests associated with those requirements would need to be performed against the verification plans, procedures, and results to show that every requirement has been verified successfully. Additionally, any defect resolutions that were designated for inclusion in the release need to be analyzed against the verification procedures and results to ensure that they were resolved and successfully verified.

Keeping track of requirements changes can be done by several methods as described in SWE-053 - Manage Requirements Changes. See also Topic 5.01 - CR-PR - Software Change Request - Problem Report

Verification is done by several methods. For guidance on inspections, see section 5.3 in the NPR. For guidance on testing, see section 4.5 in the NPR, specifically SWE-065 - Test Plan, Procedures, Reports, SWE-066 - Perform Testing, SWE-068 - Evaluate Test Results, SWE-071 - Update Test Plans and Procedures, and SWE-191 - Software Regression Testing.

For this effort, having a complete bi-directionality between requirements, design, and verification is significant and will make the analysis easier. See SWE-052 - Bidirectional Traceability. 

See also SWE-075 - Plan Operations, Maintenance, Retirement, 

3.1 Additional Guidance

Additional guidance related to this requirement may be found in the following materials in this Handbook:

3.2 Center Process Asset Libraries

See the following link(s) in SPAN for process assets from contributing Centers (NASA Only). 

4. Small Projects

No additional guidance is available for small projects.

5. Resources

5.1 References

5.2 Tools

6. Lessons Learned

6.1 NASA Lessons Learned

No Lessons Learned have currently been identified for this requirement.

6.2 Other Lessons Learned

No other Lessons Learned have currently been identified for this requirement.

7. Software Assurance

7.1 Tasking for Software Assurance

7.2 Software Assurance Products

• Approvals/sign-offs on deliveries
• List of any risks and issues found with delivery.

  
  

  Objective Evidence

  • Software test reports
  • Software traceability data
  • Software configuration management data.
  • Software assurance audit results in the change management process.
  • Software milestone results.
   Definition of objective evidence

  Objective evidence is an unbiased, documented fact showing that an activity was confirmed or performed by the software assurance/safety person(s). The evidence for confirmation of the activity can take any number of different forms, depending on the activity in the task. Examples are:

  • Observations, findings, issues, risks found by the SA/safety person and may be expressed in an audit or checklist record, email, memo or entry into a tracking system (e.g. Risk Log).
  • Meeting minutes with attendance lists or SA meeting notes or assessments of the activities and recorded in the project repository.
  • Status report, email or memo containing statements that confirmation has been performed with date (a checklist of confirmations could be used to record when each confirmation has been done!).
  • Signatures on SA reviewed or witnessed products or activities, or
  • Status report, email or memo containing a short summary of information gained by performing the activity. Some examples of using a “short summary” as objective evidence of a confirmation are:
    • To confirm that: “IV&V Program Execution exists”, the summary might be: IV&V Plan is in draft state. It is expected to be complete by (some date).
    • To confirm that: “Traceability between software requirements and hazards with SW contributions exists”, the summary might be x% of the hazards with software contributions are traced to the requirements.
  • The specific products listed in the Introduction of 8.16 are also objective evidence as well as the examples listed above.

7.3 Metrics

• # of software components (e.g., programs, modules, routines, functions, etc.) planned vs. # released in each build
• # of Software Requirements (e.g., Project, Application, Subsystem, System, etc.)
• # of software work product Non-Conformances identified by life cycle phase over time
• # of software units planned vs. # built
• # of planned software requirements implemented in each build vs. # of actual software requirements implemented in each build
• # of detailed software requirements tested to date vs. total # of detailed software requirements
• # of tests completed vs. total # of tests
• # of tests executed vs. # of tests completed
• # of Requirements tested vs. total # of Requirements
• # of Non-Conformances identified during each testing phase (Open, Closed, Severity)

See also Topic 8.18 - SA Suggested Metrics.

7.4 Guidance

For each delivery, review the delivery documentation and confirm that the document identifies the particular requirements that have been implemented for this delivery, any changes that are approved for implementation in this delivery, and any defects that need to be resolved/implemented before delivery. It is usually helpful for the project to include a list of defects or discrepancies that have not been resolved in this delivery and will either need to be resolved in a future delivery or maintenance. Delivery documentation for this information may include a delivery letter and a version description document.

The configuration management records for any changes, corrections, or new capabilities/requirements should indicate which of the change requests have been approved and the planned delivery for their implementation. Confirm that all the requirements identified for this delivery have been successfully tested by reviewing the traceability matrices to see which tests map to the requirements for the delivery and checking the test reports for those requirements.

Review the set of approved changes for the software to be delivered and confirm that all of the approved changes that were scheduled to be implemented by delivery are actually in the software deliverable. Also, confirm that: 

• The changes have been tested to confirm the approved change produces correct results,
• If the change was a defect, the changes fix the defect, and that checks have been made to catch any similar defects elsewhere in the deliverable software,
• Regression tests have been successfully run to ensure previously tested capabilities of the software still work. A full set of regression tests should be run on critical software.

For the set of defects identified in the software to be delivered, confirm that all defects planned for delivery have been resolved in one of the following ways:

• The defect is fixed and tested (confirming the above)
• The defect has an agreed-upon work-around (documented in the user documentation)
• The defect has been discussed with the customer and an agreement was reached to defer the defect to a later build or maintenance or to leave the defect as is. If the agreement is to leave the defect “as is”, identify any risk associated with this decision

Software assurance personnel should sign off on the delivery documentation signifying that the delivery is complete and correct

For guidance on testing, see section 4.5 in the NPR, specifically SWE-065 - Test Plan, Procedures, Reports, SWE-066 - Perform Testing, SWE-068 - Evaluate Test Results, SWE-071 - Update Test Plans and Procedures, and SWE-191 - Software Regression Testing. For more guidance on managing changes and release management, see SWE-053 - Manage Requirements Changes, SWE-080 - Track and Evaluate Changes, and SWE-085 - Release Management.

7.5 Additional Guidance

Additional guidance related to this requirement may be found in the following materials in this Handbook: