If the project manager does not address the issues and risks found by IV&V and track them to closure, these unaddressed risks and issues could cause the project to fail to meet their objectives (e.g. schedule, planned quality, functionality, etc.) Since IV&V personnel have generally worked across many projects, they are often likely to recognize risks and issues to the project that the project manager may not recognize.
If software-independent validation and verification (IV&V) are performed on a project, the project manager provides responses to IV&V submitted issues and risks, and track these issues and risks to closure.
IV&V is a part of Software Assurance playing a role in the NASA software risk mitigation strategy. IV&V is an objective examination of safety and mission-critical software processes and products. The key parameters for independence are technical independence, managerial independence, and financial independence. The goal of IV&V is to determine if the right system has been built and that it has been built correctly. From that perspective IV&V strives to answer the questions: will the system’s software do what it is supposed to do, will the system’s software not do what it is not supposed to do and will the system’s software respond as expected under adverse conditions.
IV&V leads to higher quality products, reduced risk, greater insight, reduced cost and knowledge transfer. IV&V also facilitates the transfer of system and software engineering best practices. IV&V status reports provide another status indicator and performance report to decision-makers (program managers). The scope of IV&V services is determined by the IV&V provider, documented in the IV&V Project Execution Plan (IPEP)
, and approved by the NASA IV&V Program. The IPEP is developed by the IV&V provider and serves as the operational document that will be shared with the project receiving IV&V support.
Once a risk or issue is submitted by IV&V, use the following considerations:
Considerations for evaluating the change and suggested solution
Project impact analysis.
Include the appropriate set of stakeholders, such as procurement, quality assurance, risk management, relevant experts, management (e.g., change requested by high visibility customer may result in a business decision to implement a change as opposed to volumes of end-users not seeing the problem), etc.
Evaluate the impact on schedule and cost, including making and testing the change and regression testing the software.
Evaluate the impact on other groups and resources, as applicable.
Evaluate the impact on functions and features, interfaces, system resource requirements.
Evaluate the impact on other baseline products, such as design, tests, documentation (traceability matrices are helpful here).
Evaluate the risk of making the change vs. not making it.
Evaluate the size, complexity, the criticality of the change.
Evaluate whether a change request is within the scope of the project.
Evaluate whether a change request is needed to meet project requirements.
Evaluate impact on performance, reliability, quality, etc.
Evaluate alternatives to making the change.
Software safety impact analysis (NASA-STD-8739.8
, NASA Software Assurance, and Software Safety Standard )
Include software quality assurance, safety personnel in this review.
Look for the potential creation of new hazard contributions and impacts.
Look for potential modification of existing hazard controls or mitigations.
Look for a detrimental effect on safety-critical software or hardware.
Determine the effect on software safety.
Determine the effect on system safety.
Capture evaluation/analysis results and related decisions, including action items.
Considerations for tracking the change
Use a change control system that is compatible with the project environment and capable of tracking change until completed.
Trace safety-critical problems back to the related system-level hazard.
Include in the tracking records the actual change request/problem reports, impact analysis, notes from evaluation/approval boards and meetings, etc.
Track the software products and versions changed as part of implementing the change (requirements, code, specifications, etc.)
Close change requests only after verification and approval of the implemented change and all associated documentation revisions.
Once the evaluation of issues and risks have been evaluated, responses are submitted to IV&V. The risks and issues are tracked until closure. The project will continue to work with IV&V in a collaborative fashion to achieve consensus on issues and risks as both are dispositioned to closure.
Additional guidance related to IV&V may be found in the following related requirement in this Handbook:
Develop, if IV&V is required on the project, IV&V findings, issues, concerns, and risks. -
Assure that the IV&V findings, issues, concerns, and risks are tracked and addressed by the project and engineering. Track findings to closure. The IV&V personnel deliver their findings, concerns, and risks to the project on a periodic basis, and generally provide a summary status at milestone reviews. SA should have access to the IV&V findings and be able to review them to assure that they are being closed out in a timely fashion. If the findings and issues are not being addressed by the project, that should be elevated up through the assurance chain of command.
Identify software risks from the IV&V issues, concerns, and risks if needed. SA documents any software risks they identify from their review of the IV&V issues, concerns, and risks
Assess why the IV&V issues, concerns, and risks were not found during the SA assessment processes. SA reviews the IV&V issues, concerns and findings and compares them to the SA findings to identify any findings that were not also identified by the SA team. Those IV&V findings not also identified by SA should be assessed to determine why they were not discovered. Assess SA processes as appropriate to enable the discovery of more of these missed findings.