1. Requirements

3.6.5 If software IV&V is performed on a project, the project manager shall provide responses to IV&V submitted issues and risks, and track these issues and risks to closure. 

1.1 Notes

NPR 7150.2, NASA Software Engineering Requirements, does not include any notes for this requirement.

1.2 History

1.3 Applicability Across Classes

Key:    - Applicable | - Not Applicable

2. Rationale

 If the project manager does not address the issues and risks found by IV&V and track them to closure, these unaddressed risks and issues could cause the project to fail to meet its objectives (e.g. schedule, planned quality, functionality, etc.) Since IV&V personnel have generally worked across many projects, they are often likely to recognize risks and issues to the project that the project manager may not recognize.

3. Guidance

If software-independent validation and verification (IV&V) are performed on a project, the project manager provides responses to IV&V submitted issues and risks and tracks these issues and risks to closure. 

IV&V is a part of Software Assurance playing a role in the NASA software risk mitigation strategy.  IV&V is an objective examination of safety and mission-critical software processes and products.  The key parameters for independence are technical independence, managerial independence, and financial independence.  The goal of IV&V is to determine if the right system has been built and that it has been built correctly.  From that perspective IV&V strives to answer the questions:  will the system’s software do what it is supposed to do, will the system’s software not do what it is not supposed to do and will the system’s software respond as expected under adverse conditions.

IV&V leads to higher quality products, reduced risk, greater insight, reduced cost, and knowledge transfer.  IV&V also facilitates the transfer of system and software engineering best practices. IV&V status reports provide another status indicator and performance report to decision-makers (program managers). The scope of IV&V services is determined by the IV&V provider, documented in the IV&V Project Execution Plan (IPEP) ⁰²⁸, and approved by the NASA IV&V Program. The IPEP is developed by the IV&V provider and serves as the operational document that will be shared with the project receiving IV&V support.

Once a risk or issue is submitted by IV&V, use the following considerations:

Considerations for evaluating the change and suggested solution

• Project impact analysis.

• Include the appropriate set of stakeholders, such as procurement, quality assurance, risk management, relevant experts, management (e.g., change requested by high visibility customer may result in a business decision to implement a change as opposed to volumes of end-users not seeing the problem), etc.
• Evaluate the impact on schedule and cost, including making and testing the change and regression testing the software.
• Evaluate the impact on other groups and resources, as applicable.
• Evaluate the impact on functions and features, interfaces, system resource requirements.
• Evaluate the impact on other baseline products, such as design, tests, documentation (traceability matrices are helpful here).
• Evaluate the risk of making the change vs. not making it.
• Evaluate the size, complexity, the criticality of the change.
• Evaluate whether a change request is within the scope of the project.
• Evaluate whether a change request is needed to meet project requirements.
• Evaluate the impact on performance, reliability, quality, etc.
• Evaluate alternatives to making the change.

• Software safety impact analysis (NASA-STD-8739.8 ²⁷⁸, NASA Software Assurance, and Software Safety Standard )

• Include software quality assurance, safety personnel in this review.
• Look for the potential creation of new hazard contributions and impacts.
• Look for potential modification of existing hazard controls or mitigations.
• Look for a detrimental effect on safety-critical software or hardware.
• Determine the effect on software safety.
• Determine the effect on system safety.

• Capture evaluation/analysis results and related decisions, including action items.

Considerations for tracking the change

• Use a change control system that is compatible with the project environment and capable of tracking change until completed.
• Trace safety-critical problems back to the related system-level hazard.
• Include in the tracking records the actual change request/problem reports, impact analysis, notes from evaluation/approval boards and meetings, etc.
• Track the software products and versions changed as part of implementing the change (requirements, code, specifications, etc.)
• Close change requests only after verification and approval of the implemented change and all associated documentation revisions.

Once the evaluation of issues and risks have been evaluated, responses are submitted to IV&V. The risks and issues are tracked until closure. The project will continue to work with IV&V collaboratively to achieve consensus on issues and risks as both are dispositioned to closure.

Additional guidance related to IV&V may be found in the following related requirement in this Handbook:

4. Small Projects

No additional guidance is available for small projects.

5. Resources

5.1 References

5.2 Tools

6. Lessons Learned

6.1 NASA Lessons Learned

No Lessons Learned have currently been identified for this requirement.

6.2 Other Lessons Learned

No other Lessons Learned have currently been identified for this requirement.

7. Software Assurance

7.1 Tasking for Software Assurance

1. Confirm that the project manager provides responses to IV&V submitted issues, findings, and risks and that the project manager tracks IV&V issues, findings, and risks to closure.

7.2 Software Assurance Products

• Records of IV&V issues and risks with status (e.g., accepted, in work, closed)

  
  

  Objective Evidence

  • Evidence that the confirmation of IV&V involvement is occurring, if applicable.
   Definition of objective evidence

  Objective evidence is an unbiased, documented fact showing that an activity was confirmed or performed by the software assurance/safety person(s). The evidence for confirmation of the activity can take any number of different forms, depending on the activity in the task. Examples are:

  • Observations, findings, issues, risks found by the SA/safety person and may be expressed in an audit or checklist record, email, memo or entry into a tracking system (e.g. Risk Log).
  • Meeting minutes with attendance lists or SA meeting notes or assessments of the activities and recorded in the project repository.
  • Status report, email or memo containing statements that confirmation has been performed with date (a checklist of confirmations could be used to record when each confirmation has been done!).
  • Signatures on SA reviewed or witnessed products or activities, or
  • Status report, email or memo containing a short summary of information gained by performing the activity. Some examples of using a “short summary” as objective evidence of a confirmation are:
    • To confirm that: “IV&V Program Execution exists”, the summary might be: IV&V Plan is in draft state. It is expected to be complete by (some date).
    • To confirm that: “Traceability between software requirements and hazards with SW contributions exists”, the summary might be x% of the hazards with software contributions are traced to the requirements.
  • The specific products listed in the Introduction of 8.16 are also objective evidence as well as the examples listed above.

7.3 Metrics

• # of software work product Non-Conformances identified by life-cycle phase over time
• # of Risks identified in each life-cycle phase (Open, Closed)
• # of Risks by Severity (e.g., red, yellow, green) over time
• # of Risks with mitigation plans vs. total # of Risks
• # of Risks trending up over time
• # of Risks trending down over time
• # of IV&V Non-Conformances/issues/risks (# Open, Closed, accepted by the project, severity, category of Non-Conformance (e.g., requirements, design, code, testing, security, etc.))
• The trend of open/closed Non-Conformances

7.4 Guidance

1. Confirm if IV&V is required on the project, see section 4.4 of NASA-STD-8739.8 ²⁷⁸ for the IV&V requirements.- See SWE-141 - Software Independent Verification and Validation
2. Develop, if IV&V is required on the project, IV&V findings, issues, concerns, and risks. -
   • Assure that the IV&V findings, issues, concerns, and risks are tracked and addressed by the project and engineering. Track findings to closure. The IV&V personnel deliver their findings, concerns, and risks to the project periodically, and generally provide a summary status at milestone reviews. SA should have access to the IV&V findings and be able to review them to assure that they are being closed out in a timely fashion. If the findings and issues are not being addressed by the project, that should be elevated up through the assurance chain of command.
   • Identify software risks from the IV&V issues, concerns, and risks if needed. SA documents any software risks they identify from their review of the IV&V issues, concerns, and risks
   • Assess why the IV&V issues, concerns, and risks were not found during the SA assessment processes. SA reviews the IV&V issues, concerns, and findings and compares them to the SA findings to identify any findings that were not also identified by the SA team. Those IV&V findings not also identified by SA should be assessed to determine why they were not discovered. Assess SA processes as appropriate to enable the discovery of more of these missed findings.