1. IV&V Surveillance in Context

This section will establish the rationale behind the creation of an IV&V Surveillance team.  It begins by describing the need for an IV&V Agent in general, and from there describes the specific circumstances that might lead NASA to stand-up an IV&V Surveillance effort for a particular project or program.  It broadly defines the need for a separate group (IV&V Surveillance) to provide insight to NASA stakeholders who have an oversight role with respect to making decisions and accepting risks for a given project or program.  Section 2.3.4 contains a ‘roadmap’ to the other sections of this topic based on the reader’s interests.

1.1 Importance of IV&V for Software Development

When software developers work on large systems like those designed for NASA, they face a substantial problem – the resulting system is ‘too big’ and ‘too complex’ for a single person to fully comprehend.  Handling this complexity (and the associated emergent behaviors, hazardous operations, and general errors in the development of the software) requires a team of competent, creative, and motivated engineers and analysts.  With enough people (and a management structure that can support them efficiently), a software developer can overcome the ‘scale’ and ‘complexity’ problems in order to build functioning software that NASA requires.  One significant problem remains, however, and the developer could never add enough people to the team to address it.

Software development teams cannot be completely objective when reviewing their own work.  The organization likely has a Verification and Validation (V&V) team or a Mission Assurance team to ensure that the original concept of the mission was realized, but these individuals are part of the team.  The software product is still ‘theirs’ in a sense, and they work with the software engineers who have a closer attachment to the product.  Being able to step outside of yourself (or your team) and consider what you could have done something wrong is at best difficult – even the idea of being wrong is uncomfortable enough, especially given the high stakes of space system software.  It is in this area that the value of an Independent Verification and Validation (IV&V) Agent becomes clear.

From the experiences of the NASA IV&V Program, even for major NASA projects like Shuttle and ISS, issues were found well into operations, despite the involvement of the Agency’s best systems engineers and software developers.  When new analysis approaches, new tools, and new perspectives are brought together to consider these large systems, new issues are discovered that could have severe consequences to the mission, including crew safety.

1.2 Role of IV&V in Software Assurance

An IV&V Agent is responsible for performing an independent assessment of the software in question.  There are various layers to this ‘independence’ (technical, managerial, cost).  One key aspect of this independence is the review of the inherent risk of the system, which results in a list of prioritized system/software capabilities.  With this risk assessment, the IV&V Agent determines which software capabilities or entities deserve the most attention.  The resulting IV&V analysis may cover the entire life cycle of the project (concept, requirements, design, code, and test) depending on the resources available.  The IV&V Agent delivers the results of these analyses to the Provider, which helps to mitigate the problem referenced in Section 1.1.  The IV&V Agent presents a more objective challenge to the software products developed by the Provider, which can lead to the identification of new issues that, once resolved, add confidence that the software will perform as expected and the mission goals will be achieved.

1.3 Acquirer Insight into IV&V Activities

For most NASA Missions, it is the responsibility of the NASA IV&V Facility to perform the IV&V analysis.  This is a useful arrangement because the Acquirer (NASA) has direct and easy access to the IV&V Agent (NASA IV&V) who is performing the work.  If there is a concern about the Provider’s products, NASA can go directly to a group that may be able to deliver some insight to support decision-making.

In some cases, however, the Provider may be given the option to select a third party to perform the IV&V analysis required by the NASA contract.  In this case, the IV&V Agent role moves outside of NASA’s direct sphere of influence.  This makes it more difficult for the Acquirer to stay informed on the results of the IV&V Agent, the associated quality of the findings, and the general health of their relationship with the Provider.  The third party, after all, is under contract to the Provider, not NASA.

The nature of IV&V efforts makes this distance between the IV&V Agent and the Acquirer concerning.  The difference in team size between the software developer and the IV&V Agent is measured in orders of magnitude, which means that the IV&V Agent cannot analyze all aspects of the software.  As mentioned in Section 1.2, IV&V performs a risk assessment that results in a prioritized list of areas for analysis.  In the interest of providing the most value to the Provider, an IV&V Agent will select the pieces of the software that they view as the riskiest.  This means that any IV&V analysis results/products relate to Computer Software Configuration Items (CSCIs) that are inherently risky or dangerous, and because the Acquirer has the responsibility to decide how much software risk they are taking on, they have an interest in the results of IV&V.

See also Topic 7.03 - Acquisition Guidance, SWE-037 - Software Milestones, SWE-039 - Software Supplier Insight, 

2. IV&V Surveillance

2.1 Purpose

In order to bridge the ‘gap’ between the Acquirer and the IV&V Agent identified in Section 1.3, NASA developed the concept of an IV&V Surveillance team.  This team interfaces with all stakeholders (Acquirer, Provider, IV&V Agent) to provide evidence that the IV&V analysis is taking place, that relationships are functional, and that findings from the IV&V Agent are reaching the right stakeholders.  Put more simply, the IV&V Surveillance team is a champion of the IV&V Agent (and all the associated expected benefits) and can make recommendations or raise concerns to enable the IV&V effort to be as effective as possible

2.2 Activities

The IV&V Surveillance team may choose from a wide variety of activities, based on the connections it is able to make and the resources available.  Typically, the surveillance effort begins by attending regular tag-ups between the Provider and the IV&V Agent, either status-focused or working discussions.  Outside of meetings, the team will carefully review the IV&V Plan to get a sense of how the IV&V Agent will conduct business.  Based on that, the IV&V Surveillance team will be able to ask more pertinent questions when things come up that appear to drift from the established plan.  Ideally, when the IV&V Agent delivers products to the Provider, the IV&V Surveillance team would be included in the distribution, and these products then trigger analysis activities to ensure that the execution of IV&V is reasonable.  If the IV&V Surveillance team is not included in the notification, they have a responsibility to ask for these products from either the Provider or IV&V Agent directly once they are aware they are available (from discussions in status tag-ups, for example).  These activities may result in any number of outputs detailed in Section 2.3.

2.3 Outputs

2.3.1 Risks

Risks are the primary output of an IV&V Surveillance effort.  They should clearly describe the potential problem along with a set of closure criteria to establish the scope of the risk.  Because an IV&V Surveillance effort does not perform technical analysis (that is why the IV&V Agent was selected), most risks focus on management or process concerns.  These types of concerns inevitably affect the technical products, so any risks should tie the underlying concern back to the higher-level safety and/or mission assurance objectives.

See also Topic 7.19 - Software Risk Management Checklists, 8.24 - Software Assurance Risk, SWE-086 - Continuous Risk Management.

2.3.2 Reports

Reports are the secondary output of an IV&V Surveillance effort and may serve as supporting evidence for the risks identified during an analysis task.  In addition to describing the objective(s), approach, results, and conclusions of the analysis, and IV&V Surveillance report may be used to fill a communications gap between the IV&V Agent and other stakeholders (see “IV&V Surveillance as Information Broker” in Section 13).  The Provider is the target of IV&V analysis products, but others may benefit from the results produced by the IV&V Agent.  In cases where stakeholders may not have easy access to IV&V products (or adequate time to review), the IV&V Surveillance reports delivered to the Acquirer can serve as a summary of IV&V findings which may prompt further investigation.

2.3.3 Issues

In some cases, an IV&V Surveillance team may observe a systemic problem with the technical analysis performed by the IV&V Agent.  This could prompt the IV&V Surveillance team to do an analysis of the software products themselves.  In this instance, the team may deliver technical issues to the Provider directly (this would have to be negotiated with the Provider, along with access to the software products).  These issues may also be used as a ‘tuning’ device to ensure that IV&V Agent analysis gets back on track.

2.3.4 Roadmap

3. The Independent Verification & Validation (IV&V) Requirements in Software Assurance and Software Safety Standard

The Independent Verification & Validation (IV&V) Requirements in Software Assurance and Software Safety Standard, NASA-STD-8739.8 ²⁷⁸, in section 4.4, are as follows:

4. Getting Started

The following sections contain topics to consider when setting up an IV&V Surveillance effort.  The success of the IV&V Surveillance effort (and also a lot of the potential problems) is based on how well the following considerations are handled.

4.1 Scope

There are several options in terms of what an IV&V Surveillance effort can attempt to cover, and the effort should be related to the concerns that prompted the Acquirer’s (or other party’s) need for additional insight into the IV&V effort.  For example, if there is a concern about the technical quality of the IV&V Agent’s work, the IV&V Surveillance team may do some IV&V analysis alongside the IV&V Agent in order to show whether the IV&V effort is meeting expectations laid out in the IV&V Plan.  In most cases, however, the IV&V Surveillance role will be limited to analyzing the IV&V products that are generated, listening to meetings, and reviewing project management plans.  The team, in this case, takes the stance that the IV&V Agent is technically competent to perform the work – the IV&V Surveillance team is then interested in the health of the IV&V Agent / Provider relationship, ensuring that the IV&V Agent has everything it needs to be successful.

See also SWE-131 - Independent Verification and Validation Project Execution Plan, Topic 8.53 - IV&V Project Execution Plan. 

4.2 Staffing

Given the domain, the first requirement of an IV&V Surveillance team member would be a systems engineering background with experience in the analysis of software development.  Beyond that, it would be helpful for an analyst to have prior experience on multiple IV&V projects as an IV&V analyst.  They should be familiar with project management documentation (plans, schedules, etc.) and its relevance to the software development life cycle.  Access to historical data on IV&V project execution is a substantial plus because it allows the team to mine for potential questions to ask, common problems, and typical analysis activities an IV&V Agents may conduct.

In addition to the technical requirements, and IV&V Surveillance analyst must be willing to do analysis at the level of project management and not at the level of software artifacts.  For analysts moving from IV&V projects to IV&V Surveillance, it can be a significant ‘culture shock,’ so be prepared to ease that transition.  This role relies on openness, creativity, and flexibility as much as it relies on knowledge of IV&V.

5. Data Access

5.1 IV&V Plan

During the initial negotiations for getting IV&V Surveillance involved, it is important to secure access to a few items.  The IV&V Plan is a necessary input before any surveillance work can be done – the IV&V Agent is bound by certain agreements and expectations covered in the plan, and any assessment must begin with those details in mind.  The IV&V Agent typically updates the IV&V Plan once per year.  Depending on the point of contact, this may be easier to get directly from the IV&V Agent. See also SWE-131 - Independent Verification and Validation Project Execution Plan, Topic 8.53 - IV&V Project Execution Plan.

5.2 Standard Meeting Materials

 Having secured the plan, the next requirement is to get invited to regular tag-ups where IV&V activities are discussed, as well as access to a repository where the meeting slides/minutes are stored.  If IV&V tag-ups are recorded, access to those would be helpful as well.

5.3 IV&V Products

The types of IV&V products required depends on how concerned IV&V Surveillance is about the technical competency of the work.  Nominally, the IV&V products that would be useful are all formal technical reports, problem reports, risks, and milestone presentations developed by the IV&V Agent.  It would be desirable for at least one member of the IV&V Surveillance team to have access to wherever these products are stored, but if that is not possible they should be on the distribution when these are delivered to the Provider (using whatever method adequately protects the sensitive nature of the document).

5.4 Relevant Standards

Many of the requirements imposed on the IV&V Agent are captured in the IV&V Plan, but there may be other sources of requirements that could assist in the development of IV&V Surveillance analysis activities.  For example, NASA projects are subject to a host of NPRs (ex. NPR 7150.2) that specify requirements for software engineering, software assurance, etc.  Depending on the terms of the contract, the Provider may be subject to those standards, or another set of “alternate standards” which have been determined to meet the intent of the NASA standards.  These standards will include requirements on both the Provider and the IV&V Agent with respect to IV&V.  It will also be important to get access to the Provider’s Software Development Plan (SDP), in order to, among other things, see what requirements the Provider has levied on the IV&V Agent.

5.4.1 IVV 09-1: Independent Verification and Validation Technical Framework

Section 15 in this topic contains a few references that may be useful when thinking about what to expect of an IV&V Agent.  Consider especially “IVV 09-1: Independent Verification and Validation Technical Framework,” which is a collection of principles and operating procedures related to delivering IV&V services.  This will not replace the need for the IV&V Agent’s IV&V Plan and other program/project-specific standards, but this could prove to be a useful benchmark for the IV&V Surveillance team.

6. Relationships

6.1 Provider

Because the IV&V Agent is under contract with the Provider, a relationship with the Provider is important for the surveillance team to build and maintain.  Access to the IV&V Agent will depend on this “gatekeeper” relationship – it may be the Provider’s decision to provide artifact and meeting access, but request that IV&V Surveillance not interact directly with the IV&V Agent.  This is less desirable, but the job of IV&V Surveillance would still be possible.  Either way, ensure that the IV&V Surveillance team engages with the Provider while minimizing disruption to the Provider.

6.2 IV&V Agent

Assuming that the Provider approves of direct communications with the IV&V Agent, this is an IV&V Surveillance team’s most valuable contact (for the obvious reason that they are doing the IV&V).  This is also the most potentially problematic relationship, for the very reason that IV&V Surveillance was conceived in the first place – the desire to not be wrong.  IV&V Surveillance could be seen in the same negative context that some Providers see the IV&V Agent (for pointing out problems or things that are “wrong”), and so special care should be taken to emphasize the shared goal of mission success.  It is also a good idea to socialize any potential risks or major findings with the IV&V Agent because they may have the information to address the concern and therefore make the risk unnecessary.

7. Products

The nature of the IV&V Surveillance products are up to the discretion of the team but should conform to the needs of the Acquirer with respect to the level of detail, format, etc.

7.1 Risks

As mentioned in Section 2.3.1, risks are the primary output of an IV&V Surveillance effort.  The team should consider any limitations on the kinds of risks they generate.  Risks may be constrained by the Acquirer in terms of applying only to Crew Safety, or only to certain CSCIs, or even only to process-based risks.  Preferably, all of these options are available, but the IV&V Surveillance team must negotiate with the Acquirer on specific needs.  Any constraints on IV&V Surveillance risks will also necessarily limit the types of activities performed.  It will also be necessary to establish a risk review board to ensure risks are clear, concise, of high quality, and conform to any Acquirer requirements.

7.2 Reports

These can be as varied as the risks, but there are a few types of reports to consider.  It would be helpful to create a report that communicates the IV&V Surveillance team’s assessment of the IV&V Plan in the context of the applicable IV&V requirements found in the governing documents, alternate standards, etc.  It would be helpful to develop a series of brief summary reports for all major deliveries from the IV&V Agent (such as a technical report).  Beyond that, consider what the Acquirer is looking for and adjust accordingly.

One other thing to consider beyond the communication of IV&V Surveillance results is the communication of IV&V results.  Just because the Provider has read the IV&V reports does not mean the Acquirer has, and even if the Acquirer has, they may not fully grasp the significance of the findings.  The IV&V Surveillance team may use its own reports as a vehicle to emphasize and magnify the key findings of the IV&V Agent to ensure that they have all the necessary support to overcome any challenges.

7.3 Briefings

Informal communications with the Acquirer may be sufficient to provide the necessary insight, but if there are standing meetings where several stakeholders meet regularly to discuss status, challenges, etc., consider using these forums as an opportunity to communicate findings and concerns about the nature of the Provider/IV&V Agent relationships.  Many groups are competing for attention at broader program tag-ups, and even software gets de-emphasized at times – so much more for the IV&V Agent.  If the organizers are willing to provide you a small slot to brief your concerns, this could be a useful way to bring software concerns to a group’s attention.

8. Objectives to Consider

The following table contains a list of potential objectives for IV&V Surveillance teams to consider and includes commentary on the importance of the question to the software assurance effort.  This list could never be exhaustive, but it is a good starting point for further objective definition.

9. Example of TRR

9.1 TRR Scenario

Suppose that the Provider has scheduled a TRR for a major CSCI and that the IV&V Agent has decided to perform an analysis of the test artifacts (for example, test cases) and the software test plan.  The IV&V Agent intends to develop a technical report to capture its analysis results/conclusions.  Given this information, what might the expected IV&V Surveillance task look like?

9.2 Entrance/Exit Criteria

Suppose that the Provider was required to adhere to NPR 7123.1B ⁰⁴¹ “NASA Systems Engineering Processes and Requirements” with respect to entrance/exit criteria for all milestones.  Taken from Appendix G of this NPR, here are some of the Success Criteria that may be applicable to the IV&V Agent’s analysis.

9.3 Questions to Consider

Because it was supposed that the IV&V Agent would analyze the software test plan and the test cases, it is easy to see how Success Criteria 1 and 9 can be supported by IV&V.  There are other questions to consider which may provide insight into these areas, and the next step for an IV&V Surveillance analyst would do generate a list of questions that will help guide the investigation.  Depending on the size of the list of questions, it might be useful to sort them into ‘focus areas,’ and this can be used as a way to break up the task (and resulting conclusion statements).  In no particular order:

10. Potential Outputs

There are many options here, but a best practice for a task of this size is to create a set of outputs for multiple levels of abstraction.

10.1 Summary Report

The most detailed output would be a summary report (~10-15 pages) that includes an Executive Summary, a purpose statement (including rationale for the surveillance task), and a collection of ‘focus areas’ which are narrative descriptions of the surveillance analysis in each area (such as “Requirements Verification” or “Configuration Management”).  Within each focus area, develop a few paragraphs talking about the expectations of IV&V Surveillance in this area, the results of the surveillance analysis activity, any difficulties, and a conclusion statement.  This is also a good place to reflect the actual findings from the IV&V Agent’s TRR Technical Report.  If the Acquirer wants to dig into the details and evidence from the surveillance analysis, this would be the appropriate place to start.

10.2 Briefing

Another output would be a PowerPoint package with the information in Section 10.1 (excluding the narrative descriptions).  Each IV&V Surveillance focus area should be identified, along with its corresponding conclusion statement.  If there are open concerns, these might be pulled from all sections into a single slide.  This output is more appropriate for a status briefing at a stakeholder meeting unless there is a significant concern about, for example, test readiness on the project (which may require a more in-depth discussion).

10.3 Summary Statement

This is probably the least useful tool for communicating IV&V Surveillance analysis results, but sometimes a summary statement may be the only avenue available, so it is important to craft statements carefully to maximize value.  Past IV&V Surveillance teams have created a small PowerPoint package (1-2 slides) to capture the Summary Statement (this is the preferred method for large Acquirer tag-ups where many organizations present status).  Consider including a short description of the initial concern, the high-level task, the most significant findings, and stakeholder value or confidence gained from this activity. 

10.4 Assurance Expectations

Given the TRR example, IV&V Surveillance should be prepared to talk about whether or not the test case analysis approach and corresponding IV&V results support the question “Is the software ready to be tested?”  It would be important to characterize whether or not the IV&V Agent has followed through on their analysis objectives, whether the supporting evidence in the report is sufficient to support their conclusions, and any explanations or rationale if objectives could not be met.  The IV&V Surveillance team should have some assessment of whether the test plan recommendations or suggestions are reasonable and reflect engineering and IV&V best practices.

11. Metrics

11.1 Monitor Information

Suppose that an IV&V Surveillance team has been invited to the bi-weekly tag-ups between the Provider and the IV&V Agent.  At this tag-up, the IV&V Agent briefs the attendees on the status of current activities, any open issues/risks, future work, and some relevant issue/risk metrics.  On the metrics slide, a chart shows the number of issues in ‘Created’ and ‘Resolved’ states (see below).

Metrics cannot tell the full story, of course, but looking at this chart, it appears that IV&V-generated issues do not linger in the ‘Created’ state for more than a month or so, which means the Provider is open to receiving and fixing these issues in a timely manner.  This would be a good sign that the IV&V Agent is able to have a positive impact on the Provider’s software.

11.2 Identify Trends

Software development is never quite so tidy, and sometimes the metrics charts can be a leading indicator.  Suppose that instead of the chart above, you saw something like the following:

It is clear that IV&V is still creating issues at a reasonably steady pace, but at the end of 2018, only a third of the issues have been resolved.  As stated in Section 11.1, metrics charts cannot tell the entire story, and there may be a valid reason why twenty issues are still unresolved.

11.3 Recognize the Problem

Now suppose that this trend continues into 2019, and the software Critical Design Review (CDR) is scheduled for February.  Given that the software CDR has likely been a target of IV&V analysis over the course of 2018, it is reasonable to assume these issues are directly relevant to whether the Provider can successfully pass CDR.  In this context, this metric looks like a leading indicator that there will be significant problems at CDR.

At this point, the IV&V Surveillance team should begin asking about the status of these issues, if it has not already come up in conversations.  It may be that the Provider intends to close them shortly and that the majority of these will disappear soon.  The team can confirm this by the presence of activity in the issue tracking tool(s) used by the Provider.  If these are actively being worked, this may not be a problem.

11.4 Raise Concerns

Depending on the degree to which the IV&V Agent is included in the overall software assurance team, this problem may not be well understood by the Provider.  Typically, the Provider assigns a single point of contact as the IV&V Liaison (in addition to their regular duties), and if this person does not transmit IV&V findings (for any number of reasons) to others in the Acquirer’s organization, this “bow wave” may come as a surprise to higher-level stakeholders.  This is where the IV&V Surveillance team can act as an information broker (see Section 13) to let those with more influence over resources and priorities know that there is a body of unresolved technical issues that should be addressed.  The IV&V Agent still has the responsibility to work with the Provider to close its issues, but the IV&V Surveillance team can play a role in facilitating that process by drawing the Acquirer’s attention to the problem.

12. Challenges

13. Lessons Learned

13.1 NASA Lessons Learned

No lessons learned have currently been identified for this topic.

13.2 Other Lessons Learned - IV&V

• Shared Understanding of IV&V Surveillance: One problem that even IV&V Agents encounter is the many disparate uses of “IV&V” and the associated meanings (see “Terminology Overload” Challenge).  If even the definition of “IV&V” is a problem, then it is much worse for a new effort like IV&V Surveillance.  If people do happen to know what IV&V is, they may erroneously assume IV&V Surveillance has a similar role, which explains why some surveillance teams have received questions from the Acquirer about the number of technical findings, or an assessment of the technical risks on the project.  Neither of these is part of the IV&V Surveillance purpose, but they are an indication that the purpose has not been well understood.  Ensuring that there is a common understanding among all stakeholders will help them ask the right questions, as well as understand exactly what kind of insight the IV&V Surveillance team can provide.

• Contractual Barriers to Information Sharing: If NASA requires insight into the IV&V effort (when a third party is chosen), and if the information-sharing requirements are muddled (see “Ambiguous IV&V Requirements”), then there is the possibility that NASA will be locked out of important insight.  IV&V Surveillance was created as a way to circumvent this problem, but unless it is backed up by specific contract language about information sharing, NASA (and, therefore, insight into mission assurance) is at the mercy of whoever happens to be the IV&V Agent and/or IV&V Liaison.

• Ambiguous IV&V Requirements: Because part of IV&V Surveillance’s role is to ensure that IV&V is being executed according to the plan (and associated requirements), there is a significant problem when those requirements are ambiguous.  For example, a requirement that states that the IV&V Agent is supposed to share products with stakeholders, but doesn’t identify those stakeholders, leaves the door open for IV&V Surveillance to be excluded from deliveries.  If an Acquirer wants insight from the IV&V Agent, it should be clearly reflected in any associated standards and contract vehicles.

• IV&V Surveillance as Information Broker: From the perspective of the IV&V Surveillance team, sometimes it seems like the IV&V Agent has everything it needs (in terms of artifact access, invitations to meetings, insight from other stakeholders), while the surveillance team is constantly asking itself “What am I missing?”  The truth is that everyone experiences these information access problems and this fact leads to an interesting function of IV&V Surveillance.  The team can act as an information broker between the stakeholders to ‘magnify’ the findings of the IV&V Agent to ensure that everyone is aware of the findings.  The IV&V Agent has a responsibility to ask for what it needs and advocate for its findings, but there are certain realities to being a third party contractor so far removed from the ultimate Acquirer.  If an IV&V Surveillance team was not performing this ‘information broker’ function, it would be worth investigating whether there is a need for it.

• “The Sky is Falling” Fallacy: Challenges such as limited artifact access and exclusion from certain meetings or tag-ups introduces an interesting problem for IV&V Surveillance teams.  In these cases, the lack of information encourages the creation of many risks that sound incredibly troubling at face value.  When information is limited, there is almost no constraint on the questions the IV&V Surveillance team might raise.  When a risk is drafted, it is helpful to ask the question “Are things really as bad as this risk makes it out to be?”  There may be another analysis activity or line of questioning that could temper the risk to something that is more reasonable (and more digestible by stakeholders).

14. Glossary

15. Resources

15.1 References

15.2 Additional Guidance

Additional guidance related to this requirement may be found in the following materials in this Handbook:

15.3 Center Process Asset Libraries

See the following link(s) in SPAN for process assets from contributing Centers (NASA Only).