2. Guidance

One of the final activities in the software development process is the qualification of flight software and hardware as an integrated system. The basis for all of the qualification efforts is the planning activity. 

The planning and process flow for the qualification of flight software can be depicted notionally by the pyramid structure of Figure 1.  The basic activities are separate actions, but the production of quality products depends on the successful implementation of all of them together. 

• The discussion in SWE-013 lays the foundation for the overall software development, with planning for the succeeding process activities of configuration management (CM), assurance, Verification and Validation (V&V), and reviews. 
• Additional guidance regarding the plans discussed in SWE-013 is available in SWE-018, SWE-102, SWE-104, and SWE-106. 
• Configuration management activities needed for the development and qualification of flight software are described in the CM plan (see SWE-103).
• Guidance for recording information, associated with qualification activities, is found in SWE-078, SWE-110, SWE-111, SWE-112, and SWE-116. 

The key feature of flight software qualification is the assessment that all necessary activities (V&V, testing, assurance, analyses, reviews, and configuration management) have been completed successfully.  One of the many review activities, the System Acceptance Review, signals the nominal completion of these flight software qualification activities.

                     Figure1. Flight Software Qualification Pyramid

The project team periodically evaluates the qualification activities to assess their correctness and degree of completion. These evaluations typically occur over a series of reviews and or related activities, and across several phases of the software development life cycle. Numerous references that discuss reviews and qualification activities, including NASA documents and external sources, are provided in the Resources section below. Two checklist tools are also included in the Resources section. One summary of a NASA approach for assessing the qualification of flight software that uses a prescribed series of reviews is described in Nelson. ³⁹⁸

• "When systems and software are ready for approval they are reviewed at the Test Readiness Review (TRR) by the internal project team. Once the software passes this internal review, it is reviewed by an independent team of engineers who have not worked on the project." ³⁹⁸

• "The independent team conducts a Flight Operational Readiness Review (ORR). When the system and software pass the ORR, the Program Manager is notified and submits the project plans and preparations to the Final Reviewer(s). The Final Reviewer(s) determines whether the software is approved for implementation or must return to software development for further work." ³⁹⁸

Another example states that the System Acceptance Review (SAR) completes the qualification activities for the flight software. The SAR is conducted by the project team to assure the software product has completed all activities prior to preparing the product for shipping.  Some consider this to be a pre-ship review. It usually occurs before the Readiness Reviews. ⁰⁴¹

Numerous reviews that occur during the software development life cycle (see SWE-019) provide interim assessments of the readiness and completeness of the software components, sub-systems, and integrated systems. These reviews generate results that all feed into the necessary qualification activities.  For many of them, 'Entrance and Success' (Exit) criteria are necessary to begin or complete the various reviews and life cycle phases.  See Topoic 7.09 - Entrance and Exit Criteria for listings and a detailed discussion of Entrance and Success criteria for the various reviews. The satisfaction of these entrance and/or success criteria in the listings provide additional confidence that the software work products are built with quality, are complete, and are ready for flight operations. The NASA Systems Engineering Processes and Requirements, NPR 7123.1A, ⁰⁴¹ and the NASA Systems Engineering Handbook,NASA SP-2007-6105, ²⁷³ provide more detailed guidance on reviews.

These formal and informal reviews are separately and collectively informed by the plans, activities, and or results gained from the software development activities.  As indicated in the Figure 1, five major process areas, when properly executed, together generate the information for the reviews.  Major activities that need to be completed to ensure the successful qualification of flight hardware are discussed in the following paragraphs. As you assess the qualification of the flight software on your project, consider the following associated 'assistance thoughts' and assess if the software is ready.

1. Planning - covers the software development on a project from inception through retirement. The processes and activities are coordinated within the project. Software needs for the project for acquisition, supply, development, operations, maintenance and retirement are developed.

During this activity, consider the following:

• Were plans reviewed for completeness?
• Were reviews planned and coordinated? 
• Were the specified Entrance and Success criteria for these reviews properly satisfied?
• Were V&V plans approved and updated as needed?
• Was appropriate planning and support provided for the configuration management and control systems that support the project?
• Were software assurance plans consistent with applicable project planning documents?

2. Configuration Management - covers the reviews and checks that all necessary items have been completed and incorporated into the product, that all identified errors and corrections have been made, that all documentation has been completed, and that all products have been evaluated, tested, and found acceptable.

During this activity, consider the following:

• Has the verification of functionality and configuration of any firmware, embedded software, and/or application flight software been addressed?
• Begin the CM process in the proposal phase when the qualification approach is defined:
  • Continue it during the design and development phase where detailed qualification planning is generated.
  • Complete it at the pre-ship/readiness review when qualification data is verified to ensure compliance with requirements.* Qualification planning is intended to be reviewed in a preliminary and final fashion at gates co-incident with PDR and CDR.
• Checklist(s) for completeness: e.g., test plans/procedures/results; issues/corrective actions; user documentation updates have been used.
• Checklist tools are intended to aid the software development team in architecting a comprehensive qualification activity, and in the identification of necessary data and reports.
• Documentation that the delivered system will perform properly in the expected operational environment?
• Was the technical data package updated to include all test results; is it complete? Does it reflect the delivered system?
• Has the completion of all analyses, examinations, and test activities' success criteria been satisfied?
• Qualification on contract and subcontract levels---prime has defined the specific technical content required in a qualification plan from the subcontractor for requirements that are flowed down to the supplier as part of the contract statement of work.
• Lists of the software components, subsystems, and integrated systems.
• Is the Certification package ready for signature and have the appropriate people who need to sign it been identified?
• Prior to pre-ship, hold a meeting to present final data to assure flight unit meets all requirements and the verification matrix is complete.

3. Software Assurance - covers the assurance planning that independently ensures conformance of software life cycle processes and products to requirements, standards, and procedures. Software assurance ensures that all processes used to acquire, develop, assure, operate and maintain the software are appropriate, sufficient, planned, reviewed, and implemented according to plan, meet any required standards, regulations, and quality requirements. Software assurance assures that the software and its related products meet their specified requirements, conform to standards and regulations, are consistent, complete, correct, safe, secure and reliable as warranted for the system and operating environment, and satisfy customer needs. Software assurance utilizes relevant project-based measurement data to monitor each product and process for possible improvements.

During this activity, consider the following:

• Were all software work products properly classified? Were classification update reviews performed?
• Were software safety criticality assessments made and agreed to between the project and the software assurance personnel?
• Was an appropriate risk assessment and management plan developed and used throughout the life cycle?
• Was oversight provided by qualification process experts and supporting subject matter experts?
• Were software assurance program metrics developed and maintained?
• Were planned assurance audits and assessments conducted?
• Have all hazards been evaluated?
• Have all risks been mitigated or accepted properly?

4. Verification & Validation (V&V) - covers the development activities that ensure that software being developed or maintained satisfies functional and other requirements and that each phase of the development process yields the right products. V&V processes determine whether the development products of a given activity conform to the requirements of that activity and whether the product satisfies its intended use and user needs. This determination may include analysis, evaluation, review, inspection, assessment, and testing of products and processes.

The dynamics of an integrated system software product demand that the V&V effort examine the correctness of the software product for each possible variation in conditions. The ability to model these complex real world conditions will be limited, however, and thus the V&V effort may have to limit itself to assessing whether the limits of the modeling are realistic and reasonable for the desired solution. Besides the project's V&V activities, Independent V&V activities may use different operators, different hardware, simulation systems, and test procedures.  They may use some developer tests or some customer generated tests. To qualify any software, a consistent logical progression of testing, in accordance with good software engineering practice, starting with unit testing and progressing to integration testing, and then to system testing, must be completed.  They need to include path and thread testing, off-nominal cases, boundary conditions, stress cases, and robustness testing. Testing produces documented evidence useful in reaching understanding and conclusions about the true characteristics of the items tested.

During this activity, consider the following:

• Were the selections of the software verification methods and criteria logical and complete?
• Were the methods applied successfully across the life cycle (e.g., software peer review/inspections procedures, re-review/inspection criteria, testing procedures)?
• Was the identification and selection of software work products to be verified logical and complete?
• Were the software verification environments that were established for the project (e.g., software testing environment, system testing environment, regression testing environment) properly accredited?
• Have the actual software verification records and analysis of the results been properly documented (e.g., test records, software peer review/inspection records) and preserved?
• Were all requirements properly verified, or otherwise treated (e.g., waived)
• Were software verification corrective actions properly documented, tracked to closure, and effectively completed?
• Were IV&V activities sufficient to reduce risk from undetected errors in the V&V activities?
• Are all requirements subjected to bi-directional traceability analysis?

5. Acceptance Review – covers the numerous acceptance activities.  One of these, the formal System Acceptance Review, verifies the completeness of the specific end products in relation to their expected maturity level and assesses compliance to stakeholder expectations. The SAR examines the system, its end products and documentation, and test data and analysis that support verification.  It also ensures that the system has sufficient technical maturity to authorize its shipment to the designated operational facility or launch site.

During this activity, consider the following:

• Were product verification and validation completed successfully?
• Were the verification and validation of products, integration of products into systems, and system verification and validation performed or witnessed by the technical team, assurance personnel, and/or customer representatives, as appropriate??
• Is the technical data package current (as-built) and complete?
• Is the transfer of certifications, warranties, or representations complete?
• Is the transfer of software products, licenses, data rights, intellectual property rights, etc., complete (SWE-085)?
• (If an acquisition) Is the technical documentation required in contract clauses complete and available? (e.g., new technology reports)
• Are the customer's acceptance criteria known and satisfied (see SWE-034)?

3. Safety

Safety considerations must be monitored in all qualification activities. Consider safety review packages as a special case of material needed for the qualification of flight software. See SWE-022 and SWE-023.

Remember that regulatory authorities will be looking for evidence that all potential hazards have been identified and that appropriate steps have been taken to deal with them.

4. Additional Guidance

Additional guidance for qualification of flight software may be found in the following applicable and inter-related requirements in this handbook:

5. Resources

5.1 Tools