bannera

Book A.
Introduction

Book B.
7150 Requirements Guidance

Book C.
Topics

Tools,
References, & Terms

SPAN
(NASA Only)

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 7 Next »

7.18 - Qualification of Flight Software
Unknown macro: {div3}

1. Purpose

This Topic provides assistance to the software development team for completing the qualification of flight software according to Agency-level requirements contained in the NPR 7150.2, NASA Software Engineering Requirements.

1.1 Introduction

Qualification of software for flight is the process of demonstrating whether the software is capable of fulfilling the specified requirements, meets the user's needs and has assured quality.  Qualification requirements are a set of criteria and conditions that have to be met in order to qualify a software product as complying with its specifications and being ready for use in its target environment.

Successful implementation of a Software Qualification Process ensures that the following items have been addressed:

  1. Criteria for the integrated software are developed that demonstrate compliance with the software and assurance requirements;
  2. Integrated software is validated using the defined criteria;
  3. Software configuration is established and controlled.
  4. Review and documentation strategies have been developed, evaluated and implemented. 
  5. Results of the qualification process have been independently reviewed.
  6. A regression strategy is developed and applied for re-testing the integrated software when a change in a software unit, subsystem, or integrated system is made.

The qualification of software for flight includes those activities related to planning, configuration management, verification, validation, software testing, software integration, system testing,   software analyses, software assurance, reviews, and acceptance. Many of the requirements in the NPR provide specific guidance for flight software qualification activities (see Table 1 in Section 4 below).  

Even though the Guidance entries for these software development activities are presented as separate essays (SWEs, Topics) in the handbook, these activities all need to be planned and completed together to assure that the qualification of the flight software is accomplished.

The NPR includes the minimum set of requirements to protect the Agency's investment in software engineering products. This Topic supplements the information in the NPR by emphasizing the key requirements affecting qualification and by providing some explanatory notes and brief suggestions for evaluating the application or intent of the requirements to the software development qualification activities.

Unknown macro: {div3}

2. Guidance

One of the final activities in the software development process is the qualification of flight software and hardware as an integrated system. The basis for all of the qualification efforts is the planning activity. 

The planning and process flow for the qualification of flight software can be depicted notionally by the pyramid structure of Figure 1.  The basic activities are separate actions, but the production of quality products depends on the successful implementation of all of them together. 

  • The discussion in [SWE-013] lays the foundation for the overall software development, with planning for the succeeding process activities of configuration management (CM), assurance, Verification and Validation (V&V), and reviews. 
  • Additional guidance regarding the plans discussed in [SWE-013] is available in [SWE-018], [SWE-102], [SWE-104], and [SWE-106]
  • Configuration management activities needed for the development and qualification of flight software are described in the CM plan (see [SWE-103]).
  • Guidance for recording information, associated with qualification activities, is found in [SWE-078], [SWE-110], SWE-111, [SWE-112], and [SWE-116]

The key feature of flight software qualification is the assessment that all necessary activities (V&V, testing, assurance, analyses, reviews, and configuration management) have been completed successfully.  One of the many review activities, the System Acceptance Review, signals the nominal completion of these flight software qualification activities.

When attempting to understand what qualification means, consider whether all the development, test, review, and acceptance activities that should have occurred actually did occur, that the results are acceptable and currently available, and that all appropriate and required documentation is complete, available, reviewed, and properly stored.


              Figure1. Flight Software Qualification Pyramid

The project team periodically evaluates the qualification activities to assess their correctness and degree of completion. These evaluations typically occur over a series of reviews and or related activities, and across several phases of the software development life cycle. Numerous references that discuss reviews and qualification activities, including NASA documents and external sources, are provided in the Resources section below. Two checklist tools are also included in the Resources section. One summary of a NASA approach for assessing the qualification of flight software that uses a prescribed series of reviews is described in Nelson, S. "Certification Processes for Safety-Critical and Mission-Critical Aerospace Software": 1

  • "When systems and software are ready for approval they are reviewed at the Test Readiness Review (TRR) by the internal project team. Once the software passes this internal review, it is reviewed by an independent team of engineers who have not worked on the project." 1
  • "The independent team conducts a Flight Operational Readiness Review (ORR). When the system and software pass the ORR, the Program Manager is notified and submits the project plans and preparations to the Final Reviewer(s). The Final Reviewer(s) determines whether the software is approved for implementation or must return to software development for further work."1

The above quotes from Nelson indicate in the terms 'internal project team', 'independent team of engineers', 'Program Manager', and 'Final Reviewers' that a high degree of collaboration is necessary to ensure a successful qualification activity.

Another example states that the System Acceptance Review (SAR) completes the qualification activities for the flight software. The SAR is conducted by the project team to assure the software product has completed all activities prior to preparing the product for shipping.  Some consider this to be a pre-ship review. It usually occurs before the Readiness Reviews.2

Numerous reviews that occur during the software development life cycle (see [SWE-019]) provide interim assessments of the readiness and completeness of the software components, sub-systems, and integrated systems. These reviews generate results that all feed into the necessary qualification activities.  For many of them, 'Entrance and Success' (Exit) criteria are necessary to begin or complete the various reviews and life cycle phases.  See Topic 7.3 for listings and a detailed discussion of Entrance and Success criteria for the various reviews. The satisfaction of these entrance and/or success criteria in the listings provide additional confidence that the software work products are built with quality, are complete, and are ready for flight operations. The NASA Systems Engineering Processes and Requirements2 and the NASA Systems Engineering Handbook3 provide more detailed guidance on reviews.

These formal and informal reviews are separately and collectively informed by the plans, activities, and or results gained from the software development activities.  As indicated in the Figure 1, five major process areas, when properly executed, together generate the information for the reviews.  Major activities that need to be completed to ensure the successful qualification of flight hardware are discussed in the following paragraphs. As you assess the qualification of the flight software on your project, consider the following associated 'assistance thoughts' and assess if the software is ready.

1. Planning - covers the software development on a project from inception through retirement. The processes and activities are coordinated within the project. Software needs for the project for acquisition, supply, development, operations, maintenance and retirement are developed.

During this activity, consider the following:

  • Were plans reviewed for completeness?
  • Were reviews planned and coordinated? 
  • Were the specified Entrance and Success criteria for these reviews properly satisfied?
  • Were V&V plans approved and updated as needed?
  • Was appropriate planning and support provided for the configuration management and control systems that support the project?
  • Were software assurance plans consistent with applicable project planning documents?

2. Configuration Management - covers the reviews and checks that all necessary items have been completed and incorporated into the product, that all identified errors and corrections have been made, that all documentation has been completed, and that all products have been evaluated, tested, and found acceptable.

During this activity, consider the following:

  • Verification of functionality and configuration of any firmware, embedded software, and/or application flight software has been addressed?
  • Begin the CM process in the proposal phase when the qualification approach is defined:
    • Continue it during the design and development phase where detailed qualification plans are generated.
    • Complete it at the pre-ship/readiness review when qualification data is verified to ensure compliance with requirements. 
  • Qualification plans are intended to be reviewed in a preliminary and final fashion at gates co-incident with PDR and CDR.
  • Checklist(s) for completeness: e.g., test plans/procedures/results; issues/corrective actions; user documentation updates have been used.
  • Checklist tools are intended to aid the qualification team in architecting a comprehensive qualification plan and data package
  • Documentation that the delivered system will perform properly in the expected operational environment?
  • Was the technical data package updated to include all test results; is it complete? Does it reflect the delivered system?
  • Has the completion of all analyses, examinations, and test activities' success criteria been satisfied?
  • Qualification on contract and subcontract levels---prime should define the specific technical content required in a qualification plan from the subcontractor for requirements that are flowed down to the supplier as part of the contract statement of work.
  • Lists of the software components, subsystems, and integrated systems.
  • Is the Certification package ready for signature and have the appropriate people who need to sign it been identified?
  • Prior to pre-ship, hold a meeting to present final data to assure flight unit meets all requirements and the verification matrix is complete.

3. Software Assurance - covers the assurance planning that independently ensures conformance of software life cycle processes and products to requirements, standards, and procedures. Software assurance ensures that all processes used to acquire, develop, assure, operate and maintain the software are appropriate, sufficient, planned, reviewed, and implemented according to plan, meet any required standards, regulations, and quality requirements. Software assurance assures that the software and its related products meet their specified requirements, conform to standards and regulations, are consistent, complete, correct, safe, secure and reliable as warranted for the system and operating environment, and satisfy customer needs. Software assurance utilizes relevant project-based measurement data to monitor each product and process for possible improvements.

During this activity, consider the following:

  • Were all software work products properly classified? Were classification update reviews performed?
  • Were software safety criticality assessments made and agreed to between the project and the software assurance personnel?
  • Was an appropriate risk assessment and management plan developed and used throughout the life cycle?
  • Was oversight provided by qualification process experts and supporting subject matter experts?
  • Were software assurance program metrics developed and maintained?
  • Were planned assurance audits and assessment conducted?
  • Have all hazards been evaluated?
  • Have all risks been mitigated or accepted properly?

4. Verification & Validation (V&V) - covers the development activities that ensure that software being developed or maintained satisfies functional and other requirements and that each phase of the development process yields the right products. V&V processes determine whether the development products of a given activity conform to the requirements of that activity and whether the product satisfies its intended use and user needs. This determination may include analysis, evaluation, review, inspection, assessment, and testing of products and processes.

The dynamics of an integrated system software product demand that the V&V effort examine the correctness of the software product for each possible variation in conditions. The ability to model these complex real world conditions will be limited, however, and thus the V&V effort may have to limit itself to assessing whether the limits of the modeling are realistic and reasonable for the desired solution. Besides the project's V&V activities, Independent V&V activities may use different operators, different hardware, simulation systems, and test procedures.  They may use some developer tests or some customer generated tests. To qualify any software, a consistent logical progression of testing, in accordance with good software engineering practice, starting with unit testing and progressing to integration testing, and then to system testing, must be completed.  They need to include path and thread testing, off-nominal cases, boundary conditions, stress cases, and robustness testing. Testing produces documented evidence useful in reaching understanding and conclusions about the true characteristics of the items tested.

During this activity, consider the following:

  • Were the selections of the software verification methods and criteria logical and complete?
  • Were the methods applied successfully across the life cycle (e.g., software peer review/inspections procedures, re-review/inspection criteria, testing procedures)?
  • Was the identification and selection of software work products to be verified logical and complete?
  • Were the software verification environments that were established for the project (e.g., software testing environment, system testing environment, regression testing environment) properly accredited?
  • Have the actual software verification records and analysis of the results been properly documented (e.g., test records, software peer review/inspection records) and preserved?
  • Were all requirements properly verified, or otherwise treated (e.g., Waived)
  • Were software verification corrective actions properly documented, tracked to closure, and effectively completed?
  • Were IV&V activities sufficient to reduce risk from undetected errors in the V&V activities?
  • Are all requirements subjected to bi-directional traceability analysis?

5. Acceptance Review – covers the numerous acceptance activities.  One of these, the formal System Acceptance Review, verifies the completeness of the specific end products in relation to their expected maturity level and assesses compliance to stakeholder expectations. The SAR examines the system, its end products and documentation, and test data and analysis that support verification.  It also ensures that the system has sufficient technical maturity to authorize its shipment to the designated operational facility or launch site.

During this activity, consider the following:

  • Were product verification and validation completed successfully?
  • Were the verification and validation of products, integration of products into systems, and system verification and validation performed or witnessed by the technical team, assurance personnel, and/or customer representatives, as appropriate??
  • Is the technical data package current (as-built) and complete?
  • Is the transfer of certifications, warranties, or representations complete?
  • Is the transfer of software products, licenses, data rights, intellectual property rights, etc., complete ([SWE-085])?
  • (If an acquisition) Is the technical documentation required in contract clauses complete and available? (e.g., new technology reports)
  • Are the customer's acceptance criteria known and satisfied (see [SWE-034])?
Unknown macro: {div3}

3. Safety

Safety considerations must be monitored in all qualification activities. Consider safety review packages as a special case of material needed for the qualification of flight software.

Remember that regulatory authorities will be looking for evidence that all potential hazards have been identified and that appropriate steps have been taken to deal with them.

Unknown macro: {div3}

4. Additional Guidance

Additional guidance for qualification of flight software may be found in the following applicable and inter-related requirements in this handbook:

Software qualification planning

See [SWE-013], [SWE-102], [SWE-104], [SWE-105], [SWE-106]

V&V activities

See [SWE-028], [SWE-029], [SWE-030], [SWE-031]

Software testing

See [SWE-066], [SWE-067], [SWE-068]

Change tracking and resolution

See [SWE-054], [SWE-080], [SWE-113]

Software classification & criticality

See [SWE-020], [SWE-132], [SWE-133]

Acceptance

See [SWE-034], [SWE-087], [SWE-088], [SWE-089], [SWE-137]

Documentation

See [SWE-078], [SWE-110], SWE-111, [SWE-112], [SWE-116]


Unknown macro: {div3}

5. References and Resources

  1. "Certification Processes for Safety-Critical and Mission-Critical Aerospace Software", Stacey Nelson, NASA CR-2003-212806, 2003
  2. NASA Systems Engineering Processes and Requirements with Change 1, NPR 7123.1A, 2009
  3. NASA Systems Engineering Handbook, NASA/SP-2007-6105 Rev1, 2007
  4. NASA Engineering and Program/Project Management Policy, NPD 7120.4D, 2010
  5. NASA Software Safety Standard, NASA STD 8719.13  (Rev B w/ Ch1 of 7/8/2004), 2004
  6. NASA Software Assurance Standard, NASA STD 8739.8, 2005
  7. IEEE Standard for Software Reviews and Audits, IEEE Std 1028-2008
  8. IEEE Standard for Software Verification and Validation, IEEE Std 1012-2004
  9. Systems and software engineering ---Content of systems and software lifecycle process information products (Documentation), ISO/IEC 15289:2006(E)
  10. Systems and software engineering – Software life cycle processes, ISO/IEC 12207, IEEE Std 12207-2008
  11. Checklist for the Contents of Software Requirements Review (SRR), 5880-CK-005-02, GSFC
  12. Requirements Peer Review Checklist, 580-CK-057-01, GSFC
  13. "Initial flight Qualification and Operational Maintenance of X-29A Flight Software", Michael R. Earls and Joel R. Sitz, NASA TM-101703, 1989
  14. Flight Unit Qualification Guidelines, John W. Welch, Aerospace Report No. TOR-2010(8591)-20, 2010


  • No labels

0 Comments