2. Implications for Models and Simulations

2.1 Implications for Embedded Models and Simulations

As noted in the previous section, embedded models and simulations are not in the scope of NASA-STD-7009 ²⁷². However, the note in NPR 7150.2 ⁰³⁹ implies that Requirements 4.4.1-4.4.6 found in NASA-STD-7009, nevertheless be addressed for such models and simulations. Each of these requirements begins with the phrase "Shall document." None of them requires that any specific activity be performed other than the relevant documentation. As the Standard states in section 4.1: "Whatever was done is to be documented, and if nothing was done a clear statement to that effect is to be documented."

2.2 Implications for Other Models and Simulations

For all other models and simulations that are deemed by the M&S Risk Assessment to be in scope of NASA-STD-7009, there is the need to ensure that the requirements of both documents are satisfied. From the perspective of NASA-STD-7009, some of the requirements in NPR-7150.2 are not related to M&S, some are supplemental to requirements in NASA-STD-7009, and others are subsets of requirements in NASA-STD-7009.

Table 1 indicates the specific relationship of each requirement in NPR 7150.2 to NASA-STD-7009. The key to the remarks in the fourth column is

• Not Related: The NPR requirement is not germane to M&S per se. It may, for example, be a requirement on the NASA Centers (cf. SWE-003: "Center Directors, or designees, shall maintain, staff, and implement a plan to continually advance the Center's in-house software engineering capability...").
• Supplemental: The NPR requirement is relevant to software for M&S, but it is over and above any requirement in NASA-STD-7009. For example, SWE-016 ("The project manager shall document and maintain a software schedule...") has no counterpart in NASA-STD-7009, which requires in Requirement 4.1.4 just that there be a plan (albeit for the M&S as a whole and not just the software), but has no requirement that this plan include a schedule.
• Subset of Requirement X: The NPR requirement is only a part of a requirement in NASA-STD-7009: Requirement 4.1.4 requires that there be a plan for the M&S that includes such software aspects as verification and configuration management, but has additional requirements on M&S-specific aspects that have no counterpart in the NPR.

3. Table 1. NPR 7150.2B vs. NASA-STD-7009

4. Rationale for STD-7009

The NASA Standard for Models and Simulations (NASA-STD-7009) ²⁷² had its genesis in the Space Shuttle Columbia Accident Investigation (2003). Generally, its purpose is to improve the "development, documentation, and operation of models and simulations" (Diaz Report) ¹⁴⁴ and per a September 2006 memo from the Office of the Chief Engineer "include a standard method to assess the credibility of the models and simulations" . After an approximately three-year development period, the NASA Standard for Models and Simulations, NASA-STD-7009 was approved by NASA's Engineering Review Board on July 11, 2008 for voluntary use.

NASA-STD-7009 holds a unique place in the world of modeling and simulation in that it is, by direction, generally applicable to all types of models and simulations (M&S) and in all phases of development, though it is primarily focused on the results of an M&S-based analysis. All standards and recommended practices for M&S to date have either been focused on a single type of M&S (e.g., structures, fluids, electrical controls, etc.) or a particular phase of M&S development (e.g., verification, validation, etc.). NASA management is confronted with numerous types of analyses that may be involved in making critical decisions. Depending on the situation at hand, a common framework for understanding the results and assessing the credibility of that analysis may seem intuitive. However, this is complicated by the vast differences in engineering systems, and, thus, the adoption of a standard like this has been slow.

After formal approval in July 2008, the NASA-STD-7009 was largely left to the individual program, project, or M&S practitioner to adopt as they wished. While already existing programs and projects were not required to adopt it, new programs and projects were to adopt it, depending on their needs, desires, and criticality of the M&S-based analysis at hand.

5. Guidance for NASA-STD-7009

Guidance for use and application of NASA-STD-7009 ²⁷² can be found in the NASA-HDBK-7009 Handbook ⁴⁶⁰ located at the following link: https://standards.nasa.gov. This is a comprehensive instruction set on the use and application of NASA-STD-7009 as it relates to Verification and Validation of Models and Simulations. Pay particular attention to the use and application of the Credibility Assessment Scale (CAS) ⁴¹⁴.

6. Resources

6.1 Tools

7. Lessons Learned

The requirements for a NASA Standard have matured from the Columbia Accident Investigation Board (CAIB) Report.  The CAIB Report found problems pertaining to "ineffective and inconsistent application of M&S tools, along with cases of misuse."  It called on NASA to "develop, validate, and maintain physics-based computer models to evaluate Thermal Protection System damage from debris impacts. These tools should provide realistic and timely estimates of any impact damage from possible debris from any source that may ultimately impact the Orbiter." NASA was to establish impact damage thresholds that trigger responsive corrective action, such as on-orbit inspection and repair, when indicated.

Lessons Learned and their applicability to the need to perform verification and validation are well documented in the above identified resources.  These should be reviewed and retained by any program or project utilizing Modeling and Simulation products throughout the Software development life cycle.

A documented lesson from the NASA Lessons Learned database notes the following:

Performance Decrease due to Propulsion Thruster Plume Impingement on the Voyager Spacecraft, Lesson Number: 0377:

"A 21% shortfall in Voyager's velocity change was suspected to be due to exhaust plume impingement. Due to the complexity of spacecraft/thruster configurations, additional care must be taken in the development and utilization of spacecraft and plume models. Analysis should be conducted on early and final designs." ⁵⁸².