1. Requirements

3.6.2 For projects reaching Key Decision Point A, the program manager shall ensure that software IV&V is performed on the following categories of projects: 

a. Category 1 projects as defined in NPR 7120.5.
b. Category 2 projects as defined in NPR 7120.5, that have Class A or Class B payload risk classification per NPR 8705.4, Risk Classification for NASA Payloads.
c. Projects selected explicitly by the Mission Directorate Associate Administrator (MDAA) to have software IV&V.

1.1 Notes

NPR 7150.2, NASA Software Engineering Requirements, does not include any notes for this requirement.

1.2 History

1.3 Applicability Across Classes

Key:    - Applicable | - Not Applicable

2. Rationale

IV&V is a technical discipline of software assurance that employs rigorous analysis and testing methodologies to identify objective evidence and conclusions to provide an independent assessment of critical products and processes throughout the software development life cycle. The evaluation of products and processes throughout the life cycle demonstrates whether the software is fit for nominal operations (required functionality, safety, dependability, etc.) and off-nominal conditions (response to faults, responses to hazardous conditions, etc.). The goal of the IV&V effort is to contribute assurance conclusions provided to the project and stakeholders based on evidence found in software development artifacts and risks associated with the intended behaviors of the software.

3. Guidance

Independent validation and verification (IV&V) is a part of Software Assurance playing a role in the NASA software risk mitigation strategy.  IV&V is an objective examination of safety and mission-critical software processes and products.  The key parameters for independence are technical independence, managerial independence, and financial independence.  The goal of IV&V is to determine if the right system has been built and that it has been built correctly.  From that perspective IV&V strives to answer the questions:  will the system’s software do what it is supposed to do, will the system’s software not do what it is not supposed to do, and will the system’s software respond as expected under adverse conditions?

IV&V leads to higher quality products, reduced risk, greater insight, reduced cost, and knowledge transfer.  IV&V also facilitates the transfer of system and software engineering best practices. IV&V status reports provide another status indicator and performance report to decision-makers (program managers).

In 2012, the Aerospace Safety Advisory Panel recommended that “NASA should establish a standard identifying the level of criticality that requires software IV&V, i.e., at what risk level must IV&V be required and therefore either be resourced or if that is not possible, a formal waiver process be in place for an accountable individual to accept the associated risk and document it.”

The Agency concurred and directed the Office of the Chief Engineer, with assistance from the Office of Safety and Mission Assurance and the NASA IV&V Program, to develop a NASA Interim Directive (NID), now incorporated into NPR 7150.2, that would replace the existing multi-step process for determining which projects must-have software IV&V with a standard identifying the level of criticality that requires software IV&V. 

The earlier IV&V is involved in a project, the better the provided service and the value of the results will be given the additional time to understand the project, its risks, and safety-critical aspects.

3.1 Expanded criteria definitions

Category 1 projects are defined in NPR 7120.5E ⁰⁸², NASA Space Flight Program, and Project Management Requirements, as human space flight projects, projects with a life cycle cost exceeding $1B, or projects with significant radioactive material.

Category 2 projects are defined in NPR 7120.5E as projects that have life cycle costs greater than $250M and less than $1B or have life cycle costs less than $250M with a high priority level based on “the importance of the activity to NASA, the extent of international participation (or a joint effort with other government agencies), the degree of uncertainty surrounding the application of new or untested technologies” and a Class A or Class B payload risk classification.

Class A payload risk classifications are defined in NPR 8705.4 ⁰⁴⁸, Risk Classification for NASA Payloads, as payloads with high priority, very low (minimized) risk, very high national significance, very high to high complexity, greater than 5-year mission lifetime, high cost, critical launch constraints, no alternative or re-flight opportunities, and/or payloads where “all practical measures are taken to achieve a minimum risk to mission success. The highest assurance standards are used.” 

Class B payload risk classifications are defined in NPR 8705.4 as payloads with high priority, low risk, high national significance, high to medium complexity, 2- 5 year mission lifetime, high to medium cost, medium launch constraints, infeasible or difficult in-flight maintenance, few or no alternative or re-flight opportunities, and/or payloads where “stringent assurance standards [are applied] with only minor compromises in application to maintain a low risk to mission success.” 

Projects selected explicitly by the Mission Directorate Associate Administrator to have software IV&V.

Per NPR 8705.4, “The importance weighting assigned to each consideration is at the discretion of the responsible Mission Directorate.”

Additional guidance related to IV&V may be found in requirement SWE-131 - Independent Verification and Validation Project Execution Plan in this Handbook.

3.2 Independent Verification & Validation (IV&V)

IV&V is a technical discipline of software assurance that employs rigorous analysis and testing methodologies to identify objective evidence and conclusions to provide an independent assessment of critical products and processes throughout the life cycle. The evaluation of products and processes throughout the life cycle demonstrates whether the software is fit for nominal operations (required functionality, safety, dependability, etc.) and off-nominal conditions (response to faults, responses to hazardous conditions, etc.). The goal of the IV&V effort is to contribute to the assurance conclusions to the project and stakeholders based on evidence found in software development artifacts and risks associated with the intended behaviors of the software. The NASA IV&V Board of Advisors supports the NASA Chief, Safety and Mission Assurance by recommending significant project needs for software IV&V beyond projects meeting the criteria in items a. and b. of SWE-141. Exceptions to the above requirement will be written by the project and responsible Center SMA organization, adjudicated by the NASA IV&V Board of Advisors, with the final decision by the NASA Chief, Safety and Mission Assurance. Additional projects, projects in other phases, or projects without a payload risk classification can be selected by the Mission Directorate Associate Administrator to be required to have software IV&V. It is NASA policy to use the NASA Independent Verification and Validation Facility as the sole provider of IV&V services when software created by or for NASA is selected for IV&V by the NASA Chief, Safety and Mission Assurance. IV&V support is funded and managed independently of the selected project. The IV&V Advisory Board will review the scope of NASA IV&V activities on an annual basis as part of the budget planning process. The scope of IV&V services is determined by the IV&V provider, documented in the IPEP, and approved by the NASA IV&V Program. The IPEP is developed by the IV&V provider and serves as the operational document that will be shared with the project receiving IV&V support.

See also Topic 8.01 - Off Nominal Testing.

Three parameters define the independence of IV&V: technical independence, managerial independence, and financial independence.

1. Technical independence requires that the personnel performing the IV&V analysis are not involved in the development of the system or its elements. The IV&V team establishes an understanding of the problem and how the system addresses the problem. Through technical independence, the IV&V team’s different perspective allows it to detect subtle errors overlooked by personnel focused on developing the system.
2. Managerial independence requires that the personnel performing the IV&V analysis are not in the same organization as the development and program management team. Managerial independence also means that the IV&V team makes its own decisions about which segments of the system and its software to analyze and test, chooses the IV&V analysis methods to apply, and defines the IV&V schedule of activities. While independent of the development and program management organization, the IV&V team provides its findings in a timely manner to both of those organizations. The submission of findings to the program management organization should not include any restrictions (e.g., requiring the approval of the development organization) or any other adverse pressures from the development group.
3. Financial independence requires that the control of the IV&V budget be vested in a group independent of the software development organization. Financial independence does not necessarily mean that the IV&V team controls the budget but that the finances should be structured so that funding is available for the IV&V team to complete its analysis or test work. No adverse financial pressure or influence is applied.

3.3 IV&V Process

The IV&V process starts early in the software development life cycle, providing feedback to the Provider organization, allowing them to modify products at optimal timeframes and in a timely fashion, thereby reducing overall project risk. The feedback also answers project stakeholders’ questions about system properties (correctness, robustness, safety, security, etc.) to make informed decisions concerning the development and acceptance of the system and its software.

The IV&V Provider performs two primary activities, often concurrently: verification and validation. Each of the activities provides a different perspective on the system/software. Verification is the process of evaluating a system and its software to provide objective evidence as to whether or not a product conforms to the build-to requirements and design specifications. Verification holds from the requirements through the design and code and into testing. Verification seeks to demonstrate that the products of a given development phase satisfy the conditions imposed at the start of or during that phase. Validation tasks seek to develop objective evidence that shows that the content of the engineering artifact is the right content for the developed system/software. The content is accurate and correct if the objective evidence demonstrates that it satisfies the system requirements (e.g., user needs, stakeholder needs, etc.), fully describes the required capability/functionality needed, and solves the right problem.

The center of the IV&V effort is on identifying and generating objective evidence that supports the correct operation of the system or refutes the correct operation of the system. The IV&V Provider typically works with the development team to understand this objective evidence, which provides artifacts such as concept studies, operations concepts, and requirements that define the overall project. The IV&V Provider uses these materials to develop an independent understanding of the project’s commitment to NASA, which forms the basis for validating lower-level technical artifacts.

3.4 Guiding Principles

Two principles help guide the development and use of objective evidence.

1. Performing IV&V throughout the entire development lifetime is the first principle; potential problems should be detected as early as possible in the development life cycle. Performing IV&V throughout the entire development lifetime provides the IV&V team with sufficient information to establish a basis for the analysis results and provides early objective evidence to the development and program management groups to help keep the development effort on track early in the life cycle.
2. The second principle is “appropriate assurance.” Given that it is not possible to provide IV&V on all aspects of a project’s software, the IV&V Provider and the project should balance risks against available resources to define an IV&V program for each project that provides IV&V so that the software will operate correctly, safely, reliably, and securely throughout its operational lifetime. The IV&V Project Execution Plan documents this tailored approach and summarizes the cost/benefit trade-offs made in the scoping process.

3.5 IV&V Requirements

The IV&V requirements are analyzed and partitioned according to the type of artifact. The requirements do not imply or require the use of any specific life cycle model. It is also important to understand that IV&V applies to any life cycle development process. The IV&V requirements document the potential scope of analysis performed by the IV&V Provider and the key responsibility of the software project to provide the information needed to perform that analysis. Additionally, scoping the IV&V analysis is according to the application of the risk assessment to determine the prioritization of activities and the level of rigor associated with performing those activities.

Software IV&V are listed below:

• The IV&V requirements apply to all IV&V efforts performed on a software development project, as tailored by the IV&V Project Execution Plan. The IV&V requirements also serve as the definition of what NASA considers IV&V. IV&V is a risk mitigation activity, and as such, the application of IV&V analysis and the rigor of that analysis is driven by the IV&V Provider’s assessment of software risk.
• The IV&V Provider shall conduct planning and risk assessments to determine the specific system/software behaviors (including the software components responsible for implementing the behaviors) to be analyzed.
• The IV&V Provider shall develop and negotiate with the project an IV&V IPEP.
• The NASA SMA Technical Authority (TA) shall review and concur with the IPEP.
• The IV&V Provider shall provide analysis results, risks, assurance statements, and data to the responsible organizations’ project management, engineering, and software assurance personnel.
  • While independent, the IV&V Provider is still a part of a project's overall safety and risk mitigation software assurance strategy. The results of IV&V analysis need to be incorporated into the overall software assurance assessment of the project and provided to the project management. IV&V Provider should support project milestone reviews and provide the project with an evaluation of the life cycle review artifacts to assist development management decisions on whether the review criteria have been met and how to proceed going forward.
• The IV&V Provider shall participate in project reviews of software activities. Participation includes providing status and results of software IV&V activities including, but not limited to, upcoming analysis activities, artifacts needed from the project, the results of the current or completed analysis, defects, and risks to stakeholders, customers, and development project personnel.
  • The most significant positive impact of IV&V analysis is when the analysis results are in phase with the development effort. Communicating defects after development artifacts are baselined increases the cost to make the changes. Additionally, the inclusion of the IV&V Provider in ongoing technical meetings keeps the IV&V Provider informed of possible changes that may affect future IV&V tasking. Supporting the ongoing technical meetings allows the IV&V Provider an opportunity to provide real-time feedback on these changes.
  • The IV&V Provider shall provide the responsible organizations’ project management, engineering, and software assurance personnel shall insight into the software IV&V and IV&V test activities. At a minimum, the IV&V Provider will be required to allow the responsible organizations’ project management, engineering, and software assurance personnel to:
• The IPEP documents the activities, methods, level of rigor, environments, tailoring (if any) of the IV&V requirements, and criteria to be used in performing verification and validation of in-scope system/software behaviors (including responsible software components) determined by the planning and scoping effort. A Provider should use a documented analysis approach to track and manage the IV&V effort aligned with ongoing development project efforts. The IPEP documents which software products are subject to which analyses and analysis requirements are wholly, partially, or not applied following the risk assessment and resource constraints. The IPEP also serves as a communication tool between the project and IV&V to set expectations for the IV&V products produced throughout the life cycle. The IPEP may require updating throughout the life cycle.
• IV&V is a focused activity that prioritizes IV&V analysis to address the highest developmental and operational software risks. IV&V priority is based on the combination of the potential for software impacts on safety and mission success and the probability factors for latent defects. IV&V analysis activities provide coverage with a degree of rigor that reflects the priority level. The initial planning and scoping effort based on the risk assessment define the starting point for the IV&V analysis. Throughout each IV&V project, continuous and iterative feedback, through the execution of analysis, identification of issues and risks, and the collection of deeper mission understanding, allows IV&V projects to “Follow The Risk” and adjust plans. The planning and scoping effort also aid in establishing the initial relationships between the IV&V Provider, the Acquirer, and the Provider.
  • Monitor the IV&V activities and plans.
  • Review the verification activities to ensure adequacy.
  • Review IV&V studies and source data.
  • Audit the software IV&V processes and practices.
  • Participate in IV&V software reviews and technical interchange meetings

• The IV&V Provider shall participate in planned software peer reviews or software inspections guided by the planning and scoping risk analysis documented in the IV&V Project Execution Plan and NASA-HDBK-2203.
• The IV&V Provider should be involved in the review/inspection process for all system/software items within the scope of their analysis.
• The IV&V Provider shall establish, record, maintain, report, and utilize IV&V management and technical measurements.
  • Note: The IV&V Provider gathers and analyzes metrics periodically to perform continuous improvement of IV&V processes and identify indicators of IV&V and project risks.
• The IV&V Provider shall assess and track software activities' actual results and performance against the software plans and identify and report any risks or findings to the responsible organizations’ project management, engineering, and software assurance personnel.
• The IV&V Provider shall track and evaluate changes to software products to evaluate for possible changes in the IV&V Provider’s risk analysis and potential adverse impacts to the software system and the development effort.
• The IV&V Provider shall assess the software development life cycle for suitability for the problem to be solved and identify and communicate any risks associated with the chosen life cycle to the responsible organizations’ project management, engineering, and software assurance personnel.
• The IV&V Provider shall identify, analyze, track, communicate, and record risks to the software and development project by NPR 8000.4, Agency Risk Management Procedural Requirements to the responsible organizations’ project management, engineering, and software assurance personnel.
• The IV&V Provider shall verify the project implements the requirements for the software listed in NPR 7150.2 and communicate any risks to the responsible organizations’ project management, engineering, and software assurance personnel.
• The IV&V Provider shall track, record, and communicate defects/issues and other results found during the execution of IV&V analysis and independent IV&V testing to the responsible organizations’ project management, engineering, and software assurance personnel.
• The IV&V Provider shall ensure that the identified defects and issues are addressed by the project.
• The IV&V Provider shall ensure that the software planned for reuse meets the fit, form, and function as a component within the new application.
• The IV&V Provider shall ensure that the system architecture contains computing-related items (subsystems, components, etc.) to fulfill the system's mission and satisfy user needs and operational scenarios or use cases.
• The IV&V Provider shall ensure that a basis for the engineering and planning of computing-related functions is the operations, mission objectives (including mission retirement), and the system.
• The IV&V Provider shall ensure that feasibility and trade studies provide the results to confidently support the critical decisions that drove the need for the study.
• The IV&V Provider shall ensure that known software-based hazard causes, contributors, and controls are identified, documented, and traced to the project requirements.
• The IV&V Provider shall ensure that known security threats and risks are identified, appropriately documented, and updated throughout the mission life cycle and communicated to the responsible organizations’ project management, engineering, and software assurance personnel.
• The IV&V Provider shall verify and validate that the in-scope software requirements and system requirements are, at a minimum, correct, consistent, complete, accurate, readable, traceable, and testable.
  • The software usually provides the interface between the user and the system hardware and the interface between system hardware components and other systems. These interfaces are critical to the successful operation and use of the system.
• The IV&V Provider shall verify and validate that the mitigations for identified security risks are in the software requirements.
  • Security is an essential aspect of any system development effort. In most systems, the software provides the primary user interface. The user interface is an element of the system that can provide undesired access. A system concept design should address known security risks through various features in the system.
• The IV&V Provider shall ensure that software requirements meet the dependability and fault tolerance required by the system.
• The IV&V Provider shall verify and validate that the relationship between the in-scope system/software requirements and the associated architectural elements is traceable correct, consistent, and complete.
  • Architectural elements are responsible for implementing specific behaviors within the software and the overall system. The interactions between these architectural elements result in the realization of the desired behaviors as well as possible undesired behaviors.
• The IV&V Provider shall verify and validate that the software architecture meets the user’s safety and mission-critical needs as defined in the requirements.
  • The architecture provides the foundation for the development of the software. It also significantly impacts how the software deals with faults and failures and how the software interfaces with the user and system components. Analysis of the architecture provides early insight into how the software is structured and how that structure can implement the requirements.
• The IV&V Provider shall verify and validate that the detailed design products are traceable, consistent, complete, accurate, and testable.
  • Detailed design is the implementation of algorithms that control and monitor the different parts of the system and allow for interaction between the system and the user and other systems. The detailed design defines how the architectural components behave to support the interactions defined in the architecture. Analysis of the detailed design includes looking at the low-level software components in the software system.
• The IV&V Provider shall verify and validate that the interfaces between the detailed design components and the hardware, users, operators, other software, and external systems are correct, consistent, complete, accurate, and testable.
  • While architecture defines the interactions between the architectural elements, each element is generally composed of lower-level components defined by the detailed design. The interfaces between these components are important in ensuring that the architectural element meets its assigned responsibilities.
• The IV&V Provider shall verify and validate that the relationship between the software requirements and the associated detailed design components is correct, consistent, and complete.
  • The detailed design components capture the approach to implementing the software requirements, including the requirements associated with fault management, security, and safety. Analysis of the relationship between the detailed design and the software requirements provides evidence that all requirements are in the detailed design.
• The IV&V Provider shall verify and validate that the software code products are consistent with architecture, complete concerning requirements, and testable.
• The IV&V Provider shall verify and validate that the software code meets the industry's best practices and software coding standards.
• The IV&V Provider shall verify and validate that the security risks in the software code are identified and mitigated.

• Note: This includes software developed by NASA, software developed for NASA, software maintained by or for NASA, COTS, GOTS, MOTS, OSS, reused software components, auto-generated code, embedded software, the software executed on processors embedded in programmable logic devices, legacy, heritage, applications, freeware, shareware, trial or demonstration software, and open-source software components. See also Topic 8.11 - Auto-Generated Code. 
• The IV&V Provider shall verify and validate the appropriate use of off-the-shelf software, including ensuring that the project has identified all open-source software used and that the security risks are identified and mitigated by the use of the open-source software.
• The IV&V Provider shall verify and validate that the project assesses the software systems for possible security vulnerabilities and weaknesses.
• The IV&V Provider shall verify and validate that the project implements the required software security risk mitigations to ensure that security objectives for software are satisfied.
• The IV&V Provider shall verify and validate the source code through analysis tools (including but not limited to static, dynamic, composition, and formal analysis tools).
  • The use of analysis tools may include the verification and validation of the analysis tools used by the development project in the process of developing the software. The results may be from static code analysis, software composition analysis, dynamic code analysis, cyclomatic complexity, or other code quality analysis tools.
• The IV&V Provider shall verify and validate that the relationship between the software design elements and the associated software units is correct, consistent, and complete.
• The IV&V Provider shall verify and validate that the relationship between software code components and corresponding requirements is correct, complete, and consistent.
  • For all of the implementation requirements, it is with code that the development of software reaches its lowest level of abstraction and that the software capabilities are implemented. Evaluating the relationship between the source code and the design components and requirements provides evidence that only the specified requirements and components are in the system. Evaluating the relationship between the source code and the design components and requirements helps minimize one aspect of the emergence of unexpected behaviors: the inclusion of behaviors not specified in the requirements. The overall analysis of the code is essential in assuring that the code does implement the required software behaviors. From a safety perspective, it is important to evaluate the code and assure that known software safety and security issues such as buffer overflows and type mismatches, among many others, are not used in safety-critical aspects of the software.
• The IV&V Provider shall verify and validate that test plans, test procedures, test cases, test environment (including simulations), and test design at all levels of testing (unit, integration, system, acceptance, etc.) are correct, complete, and consistent for verification and validation of the source code and system functions allocated to the software.
• The IV&V Provider shall verify and validate that the relationships between the test plans, test procedures, test cases, test design, source code, and system functions allocated to the software are correct, complete, and consistent.
• The IV&V Provider shall verify that the test plans, test cases, test design, and test procedures contain objective acceptance criteria that support the verification of the associated requirements for both nominal and off-nominal conditions.
• The IV&V Provider shall verify that the software test results meet the associated acceptance criteria to ensure that the software correctly implements the associated requirements.
• The IV&V Provider assesses the testing artifacts within the system’s meaning concerning the desired capabilities and expected operational system environment. The assessment includes an examination of testing at system boundary conditions to include unexpected conditions. The testing analysis assures that the project tests all requirements and that the system does what the requirements state it should do. The testing analysis also includes an analysis of the traceability information between the tests and the requirements.
• The IV&V Provider shall verify that the project tests the required software security risk mitigations to ensure that the security objectives for the software are satisfied.
• The IV&V Provider shall assess the software maintenance and operational risks concerning software elements to support the planning of IV&V activities during the maintenance phase.
  • The approach to software development on some projects results in different parts of the software going into operation at different times in the overall project life cycle. For example, a lander mission to Mars may complete the software needed for the cruise phase to Mars while continuing to work on the entry, descent, landing, and surface operations software. Some software may also have an extended lifetime such that operational updates are anytime during the operational use of the software.
  • In some cases, software anomalies cause changes to the software. IV&V is important because software changes can often have ripple effects throughout the system and cause emergent behaviors. The IV&V analysis provides insight into these possible effects and provides an overall assessment of the impact of the change.

3.6 Additional Guidance

Additional guidance related to this requirement may be found in the following materials in this Handbook:

See also Topic 8.57 - Testing Analysis. 

3.7 Center Process Asset Libraries

See the following link(s) in SPAN for process assets from contributing Centers (NASA Only). 

4. Small Projects

All projects are assessed against these criteria to determine whether they should receive IV&V services. 

5. Resources

5.1 References

5.2 Tools

6. Lessons Learned

6.1 NASA Lessons Learned

The NASA Lessons Learned database contains the following lessons learned related to IV&V:

• Independent Verification and Validation of Embedded Software (Use of IV&V Procedures). Lesson Number 723 ⁵¹⁸: "The use of Independent Verification and Validation (IV&V) processes ensures that computer software is developed by original specifications, that the software performs the functions satisfactorily in the operational mission environment for which it was designed, and that it does not perform unintended functions. Identification and correction of errors early in the development cycle are less costly than identification and correction of errors in later phases, and the quality and reliability of software are significantly improved."

• Does Software IV&V Provide Clear Benefits to NASA Projects?  Lesson Number 6656 ⁵⁸⁴: “Recent NASA spaceflight projects that have undergone IV&V of their mission software perceive the process as offering concrete benefits beyond those accrued through VV performed solely by project personnel. The specific benefits accrued to four recent JPL spaceflight projects from participation by the NASA IVV Center are discussed, and some recommendations for future IVV programs are proposed.”

6.2 Other Lessons Learned

• Software IV&V
  • IV&V approaches and resources must align with the program's risk posture.
  • A closed-loop, auditable, corrective action toolset and process must be used to manage all IV&V identified issues and risks.

7. Software Assurance

7.1 Tasking for Software Assurance

7.2 Software Assurance Products

• Issues and risks if IV&V is not being performed as required by NPR 7150.2. 

  
  

  Objective Evidence

  • Evidence that the confirmation of IV&V involvement is occurring, if applicable.
   Definition of objective evidence

  Objective evidence is an unbiased, documented fact showing that an activity was confirmed or performed by the software assurance/safety person(s). The evidence for confirmation of the activity can take any number of different forms, depending on the activity in the task. Examples are:

  • Observations, findings, issues, risks found by the SA/safety person and may be expressed in an audit or checklist record, email, memo or entry into a tracking system (e.g. Risk Log).
  • Meeting minutes with attendance lists or SA meeting notes or assessments of the activities and recorded in the project repository.
  • Status report, email or memo containing statements that confirmation has been performed with date (a checklist of confirmations could be used to record when each confirmation has been done!).
  • Signatures on SA reviewed or witnessed products or activities, or
  • Status report, email or memo containing a short summary of information gained by performing the activity. Some examples of using a “short summary” as objective evidence of a confirmation are:
    • To confirm that: “IV&V Program Execution exists”, the summary might be: IV&V Plan is in draft state. It is expected to be complete by (some date).
    • To confirm that: “Traceability between software requirements and hazards with SW contributions exists”, the summary might be x% of the hazards with software contributions are traced to the requirements.
  • The specific products listed in the Introduction of 8.16 are also objective evidence as well as the examples listed above.

7.3 Metrics

• None identified at this time

7.4 Guidance

The requirements in the IV&V section (Section 4.4) of NASA-STD-8739.8 apply to all IV&V efforts performed on a software development project.  They also serve as the definition of what NASA considers to be IV&V.  IV&V is a risk mitigation activity, and as such, the application of IV&V analysis and the rigor of that analysis is driven by the IV&V Provider’s assessment of software risk.

IV&V is a technical discipline of SA, which employs rigorous analysis and testing methodologies identifying objective evidence and conclusions to provide an independent assessment of products and processes throughout the life cycle. The independent assessment of products and processes throughout the life cycle demonstrates whether the software is fit for nominal operations (required functionality, safety, dependability, etc.), and off-nominal conditions (response to faults, responses to hazardous conditions, etc.). The goal of the IV&V effort is to provide assurance conclusions to the Project based on evidence found in software development artifacts and risks associated with the intended behaviors of the software. 

See also Topic 8.01 - Off Nominal Testing.

The IV&V Provider performs two primary activities, often concurrently: verification and validation. Each of the activities provides a different perspective on the system/software. Verification is the process of evaluating a system and its software to provide objective evidence as to whether or not a product conforms to the build-to requirements and design specifications. Verification holds from the requirements through the design and code and into testing. Verification seeks to demonstrate that the products of a given development phase satisfy the conditions imposed at the start of or during that phase. Validation tasks, on the other hand, seek to develop objective evidence that shows that the content of the engineering artifact is the right content for the developed system/software. The content is accurate and correct if the objective evidence demonstrates that it satisfies the system requirements (e.g., user needs, stakeholder needs, etc.), that it fully describes the required capability/functionality needed, and that it solves the right problem.

Software assurance should assess the IV&V activities being planned and performed against the IV&V requirements, listed below:

• The IV&V requirements apply to all IV&V efforts performed on a software development project, as tailored by the IV&V Project Execution Plan. The IV&V requirements also serve as the definition of what NASA considers IV&V. IV&V is a risk mitigation activity, and as such, the application of IV&V analysis and the rigor of that analysis is driven by the IV&V Provider’s assessment of software risk.
• The IV&V Provider shall conduct planning and risk assessments to determine the specific system/software behaviors (including the software components responsible for implementing the behaviors) to be analyzed.
• The IV&V Provider shall develop and negotiate with the project an IV&V IPEP.
• The NASA SMA Technical Authority (TA) shall review and concur with the IPEP.
• The IV&V Provider shall provide analysis results, risks, assurance statements, and data to the responsible organizations’ project management, engineering, and software assurance personnel.
  • While independent, the IV&V Provider is still a part of a project's overall safety and risk mitigation software assurance strategy. The results of IV&V analysis need to be incorporated into the overall software assurance assessment of the project and provided to the project management. IV&V Provider should support project milestone reviews and provide the project with an evaluation of the life cycle review artifacts to assist development management decisions on whether the review criteria have been met and how to proceed going forward.
• The IV&V Provider shall participate in project reviews of software activities. Participation includes providing status and results of software IV&V activities including, but not limited to, upcoming analysis activities, artifacts needed from the project, the results of the current or completed analysis, defects, and risks to stakeholders, customers, and development project personnel.
  • The most significant positive impact of IV&V analysis is when the analysis results are in phase with the development effort. Communicating defects after development artifacts are baselined increases the cost to make the changes. Additionally, the inclusion of the IV&V Provider in ongoing technical meetings keeps the IV&V Provider informed of possible changes that may affect future IV&V tasking. Supporting the ongoing technical meetings allows the IV&V Provider an opportunity to provide real-time feedback on these changes.
  • The IV&V Provider shall provide the responsible organizations’ project management, engineering, and software assurance personnel shall insight into the software IV&V and IV&V test activities. At a minimum, the IV&V Provider will be required to allow the responsible organizations’ project management, engineering, and software assurance personnel to:
• The IPEP documents the activities, methods, level of rigor, environments, tailoring (if any) of the IV&V requirements, and criteria to be used in performing verification and validation of in-scope system/software behaviors (including responsible software components) determined by the planning and scoping effort. A Provider should use a documented analysis approach to track and manage the IV&V effort aligned with ongoing development project efforts. The IPEP documents which software products are subject to which analyses and analysis requirements are wholly, partially, or not applied following the risk assessment and resource constraints. The IPEP also serves as a communication tool between the project and IV&V to set expectations for the IV&V products produced throughout the life cycle. The IPEP may require updating throughout the life cycle.
• IV&V is a focused activity that prioritizes IV&V analysis to address the highest developmental and operational software risks. IV&V priority is based on the combination of the potential for software impacts on safety and mission success and the probability factors for latent defects. IV&V analysis activities provide coverage with a degree of rigor that reflects the priority level. The initial planning and scoping effort based on the risk assessment define the starting point for the IV&V analysis. Throughout each IV&V project, continuous and iterative feedback, through the execution of analysis, identification of issues and risks, and the collection of deeper mission understanding, allows IV&V projects to “Follow The Risk” and adjust plans. The planning and scoping effort also aid in establishing the initial relationships between the IV&V Provider, the Acquirer, and the Provider.
  • Monitor the IV&V activities and plans.
  • Review the verification activities to ensure adequacy.
  • Review IV&V studies and source data.
  • Audit the software IV&V processes and practices.
  • Participate in IV&V software reviews and technical interchange meetings

• The IV&V Provider shall participate in planned software peer reviews or software inspections guided by the planning and scoping risk analysis documented in the IV&V Project Execution Plan and NASA-HDBK-2203.
• The IV&V Provider should be involved in the review/inspection process for all system/software items within the scope of their analysis.
• The IV&V Provider shall establish, record, maintain, report, and utilize IV&V management and technical measurements.
  • Note: The IV&V Provider gathers and analyzes metrics periodically to perform continuous improvement of IV&V processes and identify indicators of IV&V and project risks.
• The IV&V Provider shall assess and track software activities' actual results and performance against the software plans and identify and report any risks or findings to the responsible organizations’ project management, engineering, and software assurance personnel.
• The IV&V Provider shall track and evaluate changes to software products to evaluate for possible changes in the IV&V Provider’s risk analysis and potential adverse impacts to the software system and the development effort.
• The IV&V Provider shall assess the software development life cycle for suitability for the problem to be solved and identify and communicate any risks associated with the chosen life cycle to the responsible organizations’ project management, engineering, and software assurance personnel.
• The IV&V Provider shall identify, analyze, track, communicate, and record risks to the software and development project by NPR 8000.4, Agency Risk Management Procedural Requirements to the responsible organizations’ project management, engineering, and software assurance personnel.
• The IV&V Provider shall verify the project implements the requirements for the software listed in NPR 7150.2 and communicate any risks to the responsible organizations’ project management, engineering, and software assurance personnel.
• The IV&V Provider shall track, record, and communicate defects/issues and other results found during the execution of IV&V analysis and independent IV&V testing to the responsible organizations’ project management, engineering, and software assurance personnel.
• The IV&V Provider shall ensure that the identified defects and issues are addressed by the project.
• The IV&V Provider shall ensure that the software planned for reuse meets the fit, form, and function as a component within the new application.
• The IV&V Provider shall ensure that the system architecture contains computing-related items (subsystems, components, etc.) to fulfill the system's mission and satisfy user needs and operational scenarios or use cases.
• The IV&V Provider shall ensure that a basis for the engineering and planning of computing-related functions is the operations, mission objectives (including mission retirement), and the system.
• The IV&V Provider shall ensure that feasibility and trade studies provide the results to confidently support the critical decisions that drove the need for the study.
• The IV&V Provider shall ensure that known software-based hazard causes, contributors, and controls are identified, documented, and traced to the project requirements.
• The IV&V Provider shall ensure that known security threats and risks are identified, appropriately documented, and updated throughout the mission life cycle and communicated to the responsible organizations’ project management, engineering, and software assurance personnel.
• The IV&V Provider shall verify and validate that the in-scope software requirements and system requirements are, at a minimum, correct, consistent, complete, accurate, readable, traceable, and testable.
  • The software usually provides the interface between the user and the system hardware and the interface between system hardware components and other systems. These interfaces are critical to the successful operation and use of the system.
• The IV&V Provider shall verify and validate that the mitigations for identified security risks are in the software requirements.
  • Security is an essential aspect of any system development effort. In most systems, the software provides the primary user interface. The user interface is an element of the system that can provide undesired access. A system concept design should address known security risks through various features in the system.
• The IV&V Provider shall ensure that software requirements meet the dependability and fault tolerance required by the system.
• The IV&V Provider shall verify and validate that the relationship between the in-scope system/software requirements and the associated architectural elements is traceable correct, consistent, and complete.
  • Architectural elements are responsible for implementing specific behaviors within the software and the overall system. The interactions between these architectural elements result in the realization of the desired behaviors as well as possible undesired behaviors.
• The IV&V Provider shall verify and validate that the software architecture meets the user’s safety and mission-critical needs as defined in the requirements.
  • The architecture provides the foundation for the development of the software. It also significantly impacts how the software deals with faults and failures and how the software interfaces with the user and system components. Analysis of the architecture provides early insight into how the software is structured and how that structure can implement the requirements.
• The IV&V Provider shall verify and validate that the detailed design products are traceable, consistent, complete, accurate, and testable.
  • Detailed design is the implementation of algorithms that control and monitor the different parts of the system and allow for interaction between the system and the user and other systems. The detailed design defines how the architectural components behave to support the interactions defined in the architecture. Analysis of the detailed design includes looking at the low-level software components in the software system.
• The IV&V Provider shall verify and validate that the interfaces between the detailed design components and the hardware, users, operators, other software, and external systems are correct, consistent, complete, accurate, and testable.
  • While architecture defines the interactions between the architectural elements, each element is generally composed of lower-level components defined by the detailed design. The interfaces between these components are important in ensuring that the architectural element meets its assigned responsibilities.
• The IV&V Provider shall verify and validate that the relationship between the software requirements and the associated detailed design components is correct, consistent, and complete.
  • The detailed design components capture the approach to implementing the software requirements, including the requirements associated with fault management, security, and safety. Analysis of the relationship between the detailed design and the software requirements provides evidence that all requirements are in the detailed design.
• The IV&V Provider shall verify and validate that the software code products are consistent with architecture, complete concerning requirements, and testable.
• The IV&V Provider shall verify and validate that the software code meets the industry's best practices and software coding standards.
• The IV&V Provider shall verify and validate that the security risks in the software code are identified and mitigated.

• Note: This includes software developed by NASA, software developed for NASA, software maintained by or for NASA, COTS, GOTS, MOTS, OSS, reused software components, auto-generated code, embedded software, the software executed on processors embedded in programmable logic devices, legacy, heritage, applications, freeware, shareware, trial or demonstration software, and open-source software components. See also Topic 8.11 - Auto-Generated Code. 
• The IV&V Provider shall verify and validate the appropriate use of off-the-shelf software, including ensuring that the project has identified all open-source software used and that the security risks are identified and mitigated by the use of the open-source software.
• The IV&V Provider shall verify and validate that the project assesses the software systems for possible security vulnerabilities and weaknesses.
• The IV&V Provider shall verify and validate that the project implements the required software security risk mitigations to ensure that security objectives for software are satisfied.
• The IV&V Provider shall verify and validate the source code through analysis tools (including but not limited to static, dynamic, composition, and formal analysis tools).
  • The use of analysis tools may include the verification and validation of the analysis tools used by the development project in the process of developing the software. The results may be from static code analysis, software composition analysis, dynamic code analysis, cyclomatic complexity, or other code quality analysis tools.
• The IV&V Provider shall verify and validate that the relationship between the software design elements and the associated software units is correct, consistent, and complete.
• The IV&V Provider shall verify and validate that the relationship between software code components and corresponding requirements is correct, complete, and consistent.
  • For all of the implementation requirements, it is with code that the development of software reaches its lowest level of abstraction and that the software capabilities are implemented. Evaluating the relationship between the source code and the design components and requirements provides evidence that only the specified requirements and components are in the system. Evaluating the relationship between the source code and the design components and requirements helps minimize one aspect of the emergence of unexpected behaviors: the inclusion of behaviors not specified in the requirements. The overall analysis of the code is essential in assuring that the code does implement the required software behaviors. From a safety perspective, it is important to evaluate the code and assure that known software safety and security issues such as buffer overflows and type mismatches, among many others, are not used in safety-critical aspects of the software.
• The IV&V Provider shall verify and validate that test plans, test procedures, test cases, test environment (including simulations), and test design at all levels of testing (unit, integration, system, acceptance, etc.) are correct, complete, and consistent for verification and validation of the source code and system functions allocated to the software.
• The IV&V Provider shall verify and validate that the relationships between the test plans, test procedures, test cases, test design, source code, and system functions allocated to the software are correct, complete, and consistent.
• The IV&V Provider shall verify that the test plans, test cases, test design, and test procedures contain objective acceptance criteria that support the verification of the associated requirements for both nominal and off-nominal conditions.
• The IV&V Provider shall verify that the software test results meet the associated acceptance criteria to ensure that the software correctly implements the associated requirements.
• The IV&V Provider assesses the testing artifacts within the system’s meaning concerning the desired capabilities and expected operational system environment. The assessment includes an examination of testing at system boundary conditions to include unexpected conditions. The testing analysis assures that the project tests all requirements and that the system does what the requirements state it should do. The testing analysis also includes an analysis of the traceability information between the tests and the requirements.
• The IV&V Provider shall verify that the project tests the required software security risk mitigations to ensure that the security objectives for the software are satisfied.
• The IV&V Provider shall assess the software maintenance and operational risks concerning software elements to support the planning of IV&V activities during the maintenance phase.
  • The approach to software development on some projects results in different parts of the software going into operation at different times in the overall project life cycle. For example, a lander mission to Mars may complete the software needed for the cruise phase to Mars while continuing to work on the entry, descent, landing, and surface operations software. Some software may also have an extended lifetime such that operational updates are anytime during the operational use of the software.
  • In some cases, software anomalies cause changes to the software. IV&V is important because software changes can often have ripple effects throughout the system and cause emergent behaviors. The IV&V analysis provides insight into these possible effects and provides an overall assessment of the impact of the change.

See also SWE-178 - IV&V Artifacts, SWE-179 - IV&V Submitted Issues and Risks, 

Software assurance considers using an audit approach to determine that the IV&V provider is performing the IV&V functions defined in these requirements. See also Topic 8.12 - Basics of Software Auditing. 

See also Topic 8.57 - Testing Analysis. 

7.5 Additional Guidance

Additional guidance related to this requirement may be found in the following materials in this Handbook: