1. Purpose

This Topic provides assistance to the software development team for completing the qualification of flight software according to Agency-level requirements contained in the NPR 7150.2. Software qualification is the collection of activities performed in the software development life cycle to affirm the quality and safety of the final implemented software product. 

1.1 Introduction

Qualification of software for flight is the process of demonstrating whether the software is capable of fulfilling the specified requirements, meets the user's needs, and has assured quality. Qualification requirements are a set of criteria and conditions that have to be met in order to qualify a software product as complying with its specifications and being ready for use in its target environment. Completion of all the software qualification activities affirms the quality and safety of the final implemented software product.

Successful implementation of a Software Qualification Process ensures that the following items have been addressed:

1. Criteria for the integrated software are developed that demonstrate compliance with the software and assurance requirements.
2. Integrated software is validated using the defined criteria.
3. Software configuration is established and controlled.
4. Review and documentation strategies have been developed, evaluated, and implemented.
5. Results of the qualification process have been independently reviewed.
6. A regression strategy is developed and applied for re-testing the integrated software when a change in a software unit, subsystem, or integrated system is made.

The qualification of software for flight includes those activities related to planning, configuration management, verification, validation, software testing, software integration, system testing, software analyses, software assurance, reviews, and acceptance. Many of the requirements in the NPR provide specific guidance for flight software qualification activities (see Table 1 in Section 4 below). The execution, completion, and approval of these activities are governed by the plans and procedures that are applicable to each of the activities separately.

The NPR includes the minimum set of requirements to protect the Agency's investment in software engineering products. This Topic supplements the information in the NPR by emphasizing the key requirements affecting qualification and by providing some explanatory notes and brief suggestions for evaluating the application or intent of the requirements to the software development qualification activities. 

2. Guidance

One of the final activities in the software development process is the qualification of flight software and hardware as an integrated system. The basis for all of the qualification efforts is the planning activity. 

The planning and process flow for the qualification of flight software can be depicted notionally by the pyramid structure of Figure 1. The basic activities are separate actions, but the production of quality products depends on the successful implementation of all of them together.

• The discussion in SWE-013 lays the foundation for the overall software development, with planning for the succeeding process activities of configuration management (CM), assurance, Verification and Validation (V&V), and reviews.
• Additional guidance regarding the plans discussed in SWE-013  is available in SWE-018, and Topic 7.18 – Documentation Guidance.
• Configuration management activities needed for the development and qualification of flight software are described in the CM plan (see SCMP).
• Guidance for recording information, associated with qualification activities, is found in Topic 7.18 – Documentation Guidance  (Software Data Dictionary, Software Design Description, Interface Design Description, Software Version Description).

The key feature of flight software qualification is the assessment that all necessary activities (V&V, testing, assurance, analyses, reviews, and configuration management) have been completed successfully. One of the many review activities, the System Acceptance Review (SAR), signals the nominal completion of these flight software qualification activities.

                     Figure1. Flight Software Qualification Pyramid

The project team periodically evaluates the qualification activities to assess their correctness and degree of completion. These evaluations typically occur over a series of reviews and / or related activities, and across several phases of the software development life cycle. Numerous references that discuss reviews and qualification activities, including NASA documents and external sources, are provided in the Resources section below. Two checklist tools are also included in the Resources section. One summary of a NASA approach for assessing the qualification of flight software that uses a prescribed series of reviews is described in Nelson. ³⁹⁸

• "When systems and software are ready for approval they are reviewed at the Test Readiness Review (TRR) by the internal project team. Once the software passes this internal review, it is reviewed by an independent team of engineers who have not worked on the project." ³⁹⁸

• "The independent team conducts a Flight Operational Readiness Review (ORR). When the system and software pass the ORR, the Program Manager is notified and submits the project plans and preparations to the Final Reviewer(s). The Final Reviewer(s) determines whether the software is approved for implementation or must return to software development for further work." ³⁹⁸

Another example states that the SAR (System Acceptance Review) completes the qualification activities for the flight software. The SAR is conducted by the project team to assure the software product has completed all activities prior to preparing the product for shipping. Some consider this to be a pre-ship review. It usually occurs before the Readiness Reviews. ⁰⁴¹

Numerous reviews that occur during the software development life cycle (see SWE-019) provide interim assessments of the readiness and completeness of the software components, sub-systems, and integrated systems. These reviews generate results that all feed into the necessary qualification activities. For many of them, 'Entrance and Success' (Exit) criteria are necessary to begin or complete the various reviews and life-cycle phases. See Topic 7.09 - Entrance and Exit Criteria for listings and a detailed discussion of Entrance and Success criteria for the various reviews. The satisfaction of these entrance and/or success criteria in the listings provide additional confidence that the software work products are built with quality, are complete, and are ready for flight operations. The NASA Systems Engineering Processes and Requirements, NPR 7123.1,  ⁰⁴¹ and the NASA Systems Engineering Handbook, NASA SP-2007-6105, ²⁷³ provide more detailed guidance on reviews.

These formal and informal reviews are separately and collectively informed by the plans, activities, and or results gained from the software development activities. As indicated in the Figure 1, five major process areas, when properly executed, together generate the information for the reviews. Major activities that need to be completed to ensure the successful qualification of flight hardware are discussed in the following paragraphs. As you assess the qualification of the flight software on your project, consider the following associated 'assistance thoughts' and assess if the software is ready.

1. Planning - covers the software development on a project from inception through retirement. The processes and activities are coordinated within the project. Software needs for the project for acquisition, supply, development, operations, maintenance, and retirement are developed..

During this activity, consider the following:

• Were plans reviewed for completeness?
• Were reviews planned and coordinated?
• Were the specified Entrance and Success criteria for these reviews properly satisfied?
• Were V&V plans approved and updated as needed?
• Was appropriate planning and support provided for the configuration management and control systems that support the project?
• Were software assurance plans consistent with applicable project planning documents? 

2. Configuration Management - covers the reviews and checks that all necessary items have been completed and incorporated into the product, that all identified errors and corrections have been made, that all documentation has been completed, and that all products have been evaluated, tested, and found acceptable.

During this activity:

• Address the verification of functionality and configuration of any firmware, embedded software, and/or application flight software.
• Begin the CM (Configuration Management) process in the proposal phase when the qualification approach is defined:
  
  • Continue it during the design and development phase where detailed qualification plans are generated.
  • Complete it at the pre-ship/readiness review when qualification data is verified to ensure compliance with requirements.
• Review qualification planning in a preliminary and final fashion at gates co-incident with PDR and CDR.
• Use checklist(s) to verify completeness (e.g., include test plans/procedures/results; issues/corrective actions; user documentation updates).
• Use checklist tools to aid the software development team in architecting a comprehensive qualification activity and to identify necessary data and reports.
• Include documentation that the delivered system will perform properly in the expected operational environment.
• Verify that the technical data package is updated to include all test results, is complete, and reflects the delivered system.
• Satisfy the success criteria for the completion of all analyses, examinations, and test activities.
• Verify the qualification on contract and subcontract levels—specifically, that the prime has defined the specific technical content required in a qualification plan from the subcontractor for requirements that are flowed down to the supplier as part of the contract statement of work.
• Include lists of the software components, subsystems, and integrated systems.
• Verify that the Certification package is ready for signature and that the appropriate people who need to sign it have been identified.
• Prior to pre-ship, hold a meeting to present final data to assure flight unit meets all requirements and the verification matrix is complete.

3. Software Assurance - covers the assurance planning that independently ensures conformance of software life cycle processes and products to requirements, standards, and procedures. Software assurance ensures that all processes used to acquire, develop, assure, operate and maintain the software are appropriate, sufficient, planned, reviewed, and implemented according to plan, meet any required standards, regulations, and quality requirements. Software assurance assures that the software and its related products meet their specified requirements; conform to standards and regulations; are consistent, complete, correct, safe, secure, and reliable as warranted for the system and operating environment; and satisfy customer needs. Software assurance utilizes relevant project-based measurement data to monitor each product and process for possible improvements.

During this activity, consider the following:

• Were all software work products properly classified? Were classification update reviews performed?
• Were software safety criticality assessments made and agreed to between the project and the software assurance personnel?
• Was an appropriate risk assessment and management plan developed and used throughout the life cycle?
• Was oversight provided by qualification process experts and supporting subject matter experts?
• Were software assurance program metrics developed and maintained?
• Were planned assurance audits and assessments conducted?
• Have all hazards been evaluated?
• Have all risks been mitigated or accepted properly?

4. Verification & Validation (V&V) - covers the development activities that ensure that software being developed or maintained satisfies functional and other requirements and that each phase of the development process yields the right products. V&V processes determine whether the development products of a given activity conform to the requirements of that activity and whether the product satisfies its intended use and user needs. This determination may include analysis, evaluation, review, inspection, assessment, and testing of products and processes.

The dynamics of an integrated system software product demand that the V&V effort examine the correctness of the software product for each possible variation in conditions. The ability to model these complex real world conditions will be limited, however, and thus the V&V effort may have to limit itself to assessing whether the limits of the modeling are realistic and reasonable for the desired solution. Besides the project's V&V activities, Independent V&V activities may use different operators, different hardware, simulation systems, and test procedures. They may use some developer tests or some customer generated tests. To qualify any software, a consistent logical progression of testing—in accordance with good software engineering practice, starting with unit testing, progressing to integration testing, and then to system testing—must be completed. They need to include path and thread testing, off-nominal cases, boundary conditions, stress cases, and robustness testing. Testing produces documented evidence useful in reaching understanding and conclusions about the true characteristics of the items tested.

During this activity, consider the following:

• Were the selections of the software verification methods and criteria logical and complete?
• Were the methods applied successfully across the life cycle (e.g., software peer review/inspections procedures, re-review/inspection criteria, testing procedures)?
• Was the identification and selection of software work products to be verified logical and complete?
• Were the software verification environments that were established for the project (e.g., software testing environment, system testing environment, regression testing environment) properly accredited?
• Have the actual software verification records and analysis of the results been properly documented (e.g., test records, software peer review/inspection records) and preserved?
• Were all requirements properly verified or otherwise treated (e.g., waived)?
• Were software verification corrective actions properly documented, tracked to closure, and effectively completed?
• Were IV&V activities sufficient to reduce risk from undetected errors in the V&V activities?

Are all requirements subjected to bi-directional traceability analysis?

5. Acceptance Review – covers the numerous acceptance activities. One of these, the formal System Acceptance Review, verifies the completeness of the specific end products in relation to their expected maturity level and assesses compliance to stakeholder expectations. The SAR examines the system, its end products and documentation, and test data and analysis that support verification. It also ensures that the system has sufficient technical maturity to authorize its shipment to the designated operational facility or launch site.

During this activity, consider the following:

• Were product verification and validation completed successfully?
• Were the verification and validation of products, integration of products into systems, and system verification and validation performed or witnessed by the technical team, assurance personnel, and/or customer representatives, as appropriate??
• Is the technical data package current (as-built) and complete?
• Is the transfer of certifications, warranties, or representations complete?
• Is the transfer of software products, licenses, data rights, intellectual property rights, etc., complete (SWE-085)?
• (If an acquisition) Is the technical documentation required in contract clauses complete and available (e.g., new technology reports)?
• Are the customer's acceptance criteria known and satisfied (see SWE-034)?

3. Safety

Safety considerations must be monitored in all qualification activities. Consider safety review packages as a special case of material needed for the qualification of flight software. See SWE-022 and SWE-023.

Remember that regulatory authorities will be looking for evidence that all potential hazards have been identified and that appropriate steps have been taken to deal with them.

4. Additional Guidance

Additional guidance for qualification of flight software may be found in the following topic and applicable and inter-related requirements in this Handbook:

5. Resources

5.1 Tools