1. Requirements

3.6.1 The project manager shall plan and implement software assurance per NASA-STD-8739.8. 

1.1 Notes

Software assurance activities occur throughout the life of the project. Some of the actual analyses and activities may be performed by engineering or the project. Software Assurance directions, requirements, and guidance can be found in the NASA-STD-8739.8, and the Software Engineering Handbook.

1.2 History

1.3 Applicability Across Classes

Key:    - Applicable | - Not Applicable

2. Rationale

To:

1. Ensure that the software processes, software procedures, and software products used to produce and sustain the software conform to all requirements and standards specified to govern those software processes, software procedures, and software products.
2. Ensure that the software systems are safe and that the software safety-critical requirements and processes are followed.
3. Ensure that the software systems are secure.

The software assurance process is the planned and systematic set of activities that ensure the conformance of software life cycle processes and products to requirements, standards, and procedures.  Software assurance assures that the software and its related products meet their specified requirements, conform to standards and regulations, are consistent, complete, correct, safe, secure, and reliable as warranted for the system and operating environment, and satisfy customer needs.

3. Guidance

The purpose of the Software Assurance and Software Safety Standard is to define the requirements to implement a systematic approach to Software Assurance (SA), Software Safety, and IV&V for software created, acquired, provided, or maintained by or for NASA. Various personnel in the program, project, or facility, and Safety and Mission Assurance (SMA) organizations can perform the activities required to satisfy these requirements. The Software Assurance and Software Safety Standard provides a basis for personnel to perform software assurance, software safety, and IV&V activities consistently throughout the life of the software, that is, from its conception, through creation to operations and maintenance, and until the software is retired.

This includes products delivered to and used within NASA, and products developed and acquired by NASA. Software assurance assists in risk mitigation by helping expose potential defects in products and processes, thus preventing problems from evolving.  However, it also, through its metrics, tracking, and analysis activities, enables the improvement of future products and services.  Software assurance often serves as the corporate memory from project to project, sharing potential problem areas and lessons learned.  

Software assurance is the level of confidence that software is free from vulnerabilities, either intentionally designed into the software or accidentally inserted at any time during its lifecycle, and that the software functions in an intended manner and that the software does not function in an unintended manner. The objectives of the Software Assurance and Software Safety Standard include:

a. Ensuring that the processes, procedures, and products used to produce and sustain the software conform to all requirements and standards specified to govern those processes, procedures, and products.

b. Ensuring that the software systems are safe and that the software safety-critical requirements and processes are followed.

c. Ensuring that the software systems are secure.

The application and approach to meeting the Software Assurance and Software Safety activities will vary based on the system and software products and processes to which they are applied. The Software Assurance and Software Safety Standard stresses coordination between the software assurance sub-disciplines, as well as with system safety, system reliability, hardware quality, system security, and software engineering, to maintain the system perspective and minimize duplication of effort.

Project and SMA Management support of the Software Assurance function is essential for Software Assurance processes to be effective. The Software Assurance support minimally includes:

(1) Management is familiar with and understands the Software Assurance function’s purposes, concepts, practices, and needs.

(2) Management provides the Software Assurance function with an appropriate level of skilled resources (people, equipment, knowledge, methods, facilities, and tools) to accomplish their project responsibilities.

(3) Management receives and acts upon information provided by the Software Assurance function throughout a project.

Software engineering and the software assurance disciplines are integrally related and yet each has its responsibilities.  Jointly they are responsible for providing project management with the optimal solution for software to meet the engineering, safety, quality, and reliability needs of the project.  This necessitates a close working relationship to establish the appropriate levels of effort for both.  The NASA Procedural Requirements, NPR 7150.2 ⁰⁸³, NASA Software Requirements invokes the NASA Software Assurance and Software Safety Standard (NASA-STD-8739.8 ²⁷⁸), requiring a close working relationship, understanding of roles and responsibilities, and establishing expected communication paths.   NPR 7150.2, besides laying out the NASA minimum requirements for software development, provides the NASA software classification upon which software engineering, software assurance, and software safety all base their tailoring.  

Software assurance includes the disciplines of software assurance, software quality, software safety, and Independent Verification & Verification (IV&V). All of these activities work together to ensure high-quality products that operate safely. The amount of software assurance required for a project is based on the classification of the software SWE-020 and the risk associated with the project and the software.

NASA's Safety and Mission Assurance organizations assure that the products and processes are implemented according to the agreed-upon plan(s). Software assurance is recommended on software activities and products, including solicitations, contracts, and memorandums of agreements, software plans, requirements, design, implementation, verification, validation, certification, acceptance, maintenance, operations, and retirement activities.

To avoid misunderstandings and issues once the project has begun, the following software assurance items need to be considered for inclusion in the provider's contract (see Topic 7.03 - Acquisition Guidance):

• Access to provider processes and documentation, see SWE-040.
• Software assurance activities, meetings, reviews, etc. planned for the project.
• Standards and requirements for provider-performed software assurance activities.
• Software assurance metrics are to be collected and reported by the software provider.

Use checklists, documented procedures, templates, etc. to ensure consistency among software assurance activities across projects.

The team is to implement the software engineering products and processes according to the plans established by the project. Software assurance activities ensure this happens through a series of reviews and audits. For NASA, NASA-STD-8739.8 captures the software assurance and software safety requirements in a single place for more uniform applications across projects where software is developed or acquired for NASA.

NASA-specific software assurance resources are available in Software Processes Across NASA (SPAN), accessible to NASA users from the SPAN tab in this Handbook. 

4. Small Projects

Software assurance is required regardless of project size. Since NASA-STD-8739.8 ²⁷⁸ defines roles, not individuals, projects with limited personnel resources, can use one person to fulfill multiple roles and perform multiple software assurance functions, as long as the proper independence for the specific requirement is maintained. In small project situations, sometimes it will be necessary for a project to have software assurance conducted by someone not on the project, but potentially from the same organization. This is not the preferred approach, but better than having no software assurance done at all.

Additionally, for acquired software, software assurance is performed by both the acquirer and the provider, so projects with small acquirer staffs could consider doing more "insight" (Surveillance mode requiring the monitoring of customer identified metrics and contracted milestones) per NASA-STD-8739.8, "Tailoring the implementation of software assurance requirements is acceptable commensurate with the program/project classification as well as size, complexity, criticality, and risk."

5. Resources

5.1 References

5.2 Tools

6. Lessons Learned

6.1 NASA Lessons Learned

No Lessons Learned have currently been identified for this requirement.

6.2 Other Lessons Learned

• Software quality assurance implementation is a balancing activity that must be tailored as project appropriate ⁴⁴⁴: (Lesson 2 of 12)

Software Quality Assurance engineers must determine, based on experience, what trade-offs are to be made within the specific project since each project has different objectives. Gilles defined some of these relationships between the quality assurance criteria e.g., an inverse relationship between usability and efficiency; a direct relationship between maintainability and reusability ². To make the best trade-offs, all relationships must be understood and experience used to interpret the impact."

• Software quality assurance (SQA) must evaluate the process as well as the products ⁴⁴⁴: (Lesson 3 of 12)

Within NASA, SA tends to focus on the products produced, such as the requirements documents, design, code listing, and test plans. But the focus of SA is to monitor continuously throughout the Software Development Life Cycle to ensure the quality of the delivered products. This requires monitoring both the processes and the products. In process assurance, SA provides management with objective feedback regarding compliance with approved plans, procedures, standards, and analyses. Product assurance activities focus on the changing level of product quality within each phase of the life cycle, such as the requirements, design, code, and test plan. The objective is to identify and eliminate defects throughout the life cycle as early as possible, thus reducing test and maintenance costs.

• There must be a software assurance plan ⁴⁴⁴: (Lesson 4 of 12)

The plan will specify the goals, what is to be performed, standards against which the development work is to be measured, all relevant procedures, and the organizational structure of the Quality Assurance within the project. 

• Software quality assurance must span the entire software development life cycle ⁴⁴⁴: (Lesson 5 of 12)

Software Development starts at the concept phase and continues through maintenance. SA activities and funding should also start at the concept definition and continue through the entire life cycle. At each phase, some activities could be performed ⁴. The project and the quality assurance office must work together to negotiate what activities are appropriate at each phase based on risks and resources. 

• Requirements are the birthplace of successful projects ⁴⁴⁴: (Lesson 6 of 12)

Although SA is performed across the entire life cycle, the success of a project can often be determined by the attention paid to requirements. The earlier in the life cycle potential risks are identified, the easier it is to eliminate or at least manage the conditions that introduce that risk. Problems not found until the testing phase are approximately 14 times more costly to fix than if found and fixed earlier in the requirements phase. The first tangible representation of the capabilities produced is the requirements specification document, be it system, hardware, software, or operational requirements. The document also serves to establish the basis for all the project's engineering management and assurance functions. If the quality of the requirements specification is poor, the project is at risk even before work begins.⁷ Therefore, a specific lesson in SA is the emphasis on requirements.

• Software quality assurance does not equal testing ⁴⁴⁴: (Lesson 7 of 12)

From the aspect of quality assurance, the purpose of testing is to:

• • • Assure problems are documented, corrected, and used for process improvement.
    • Assure problem reports are assessed for their validity.
    • Assure reported problems and their associated corrective actions are implemented following customer-approved solutions.
    • Provide feedback to the developer and the user of problem status.
    • Provide data for measuring and predicting Software Quality and Software Reliability."

• Metrics are a necessity ⁴⁴⁴: (Lesson 8 of 12)

Software Metrics are often ignored during the early Software Development Life Cycle phases and not actively associated with SA - but should be. For SA practitioners, with their responsibility for assuring both the processes and the products of the Software Development, measurement is critical. Throughout each of the life cycle phases metrics have proven invaluable in the evaluation of the quality of the products and processes ⁵.

• Safety and reliability are critical aspects of SQA ⁴⁴⁴: (Lesson 9 of 12)

Safety is a team effort and everyone's responsibility. Software within NASA is a vital part of the system and can impact the safety of the mission. Project managers, Systems Engineers, Software Leads and engineers, Software Assurance or Quality Assurance, and System Safety Personnel all play a part in creating a safe system. Reliability and safety cannot be tested at the end of a project; they must be built in as the software is being Developed.
Reliability impacts safety - a system cannot be deemed safe if it is not reliable.

• Independent verification and validation (IV&V) is an important tool within SQA ⁴⁴⁴: (Lesson 10 of 12)

NASA defines Software IV&V as a Systems Engineering Process employing rigorous methodologies for evaluating the correctness and quality of the software product throughout the Software Development Life Cycle.  

• Hardware does not equal software ⁴⁴⁴! (Lesson 11 of 12)

The influence of Hardware Quality Assurance is evident in the Software Quality Assurance practitioner community, where hardware-intensive systems and typical hardware-related concerns predominate. Two issues that dominate discussions about hardware are time and operating conditions. Software, however, is built with different constraints and considerations. 

• Risk Management is not optional ⁴⁴⁴: (Lesson 12 of 12)

Risk is a daily reality on all projects, and Continuous Risk Management should become just as routine. It should be ongoing and comfortable and neither imposed nor forgotten. Like any good habit, it should seamlessly fit into daily work.

Software Risk Management is important because it helps avoid disasters, rework, and overkill. The objectives of Software Risk Management are to identify, address, and eliminate software risk items before they become threats to success or major sources of rework. Good project managers are also good managers of risk. It makes good business sense for all software development projects to incorporate risk management as part of project management.

7. Software Assurance

7.1 Tasking for Software Assurance

1. Perform according to the software assurance plan and the software assurance and software safety standard requirements in NASA-STD-8739.8.  ²⁷⁸

7.2 Software Assurance Products

• Evidence showing that the project has implemented NASA-STD-8739.8.

• Software Assurance Process Audit Report
• Audit results on the requirements of NASA-STD 8739.8, including findings and corrective action plans.

  
  

  Objective Evidence

  • NPR 7150.2 and NASA-STD-8739.8 requirements mapping matrices signed by the engineering and SMA technical authorities for each development organization.
   Definition of objective evidence

  Objective evidence is an unbiased, documented fact showing that an activity was confirmed or performed by the software assurance/safety person(s). The evidence for confirmation of the activity can take any number of different forms, depending on the activity in the task. Examples are:

  • Observations, findings, issues, risks found by the SA/safety person and may be expressed in an audit or checklist record, email, memo or entry into a tracking system (e.g. Risk Log).
  • Meeting minutes with attendance lists or SA meeting notes or assessments of the activities and recorded in the project repository.
  • Status report, email or memo containing statements that confirmation has been performed with date (a checklist of confirmations could be used to record when each confirmation has been done!).
  • Signatures on SA reviewed or witnessed products or activities, or
  • Status report, email or memo containing a short summary of information gained by performing the activity. Some examples of using a “short summary” as objective evidence of a confirmation are:
    • To confirm that: “IV&V Program Execution exists”, the summary might be: IV&V Plan is in draft state. It is expected to be complete by (some date).
    • To confirm that: “Traceability between software requirements and hazards with SW contributions exists”, the summary might be x% of the hazards with software contributions are traced to the requirements.
  • The specific products listed in the Introduction of 8.16 are also objective evidence as well as the examples listed above.

7.3 Metrics

• The number of audit findings per audit, including the number of findings from process non-compliances and process maturity.
• # of software work product Non-Conformances identified by life-cycle phase over time

7.4 Guidance

Step 1 Determine the Software Assurance plan and the Software Assurance and Software Safety Standard requirements mapping per the software classification and the engineering requirements mapping matrix to verify the methods to fulfill the requirements in NASA-STD-8739.8.

Step 2 Perform an audit, at least every two years or upon request, to verify the software assurance and software safety requirements are being followed and that the products and metrics are being produced and provided to the engineering and project office. 

Step 3 Monitor issues, corrective actions, findings to closure- assess the engineering, contractor, software assurance, and software safety issues, corrective actions, finding to determine if the items have been addressed.

The software assurance process is the planned and systematic set of activities that ensure the conformance of software life cycle processes and products to requirements, standards, and procedures.  Software assurance assures that the software and its related products meet their specified requirements, conform to standards and regulations, are consistent, complete, correct, safe, secure, and reliable as warranted for the system and operating environment, and satisfy customer needs.

1. Assess the Software Assurance plan and the Software Assurance and Software Safety Standard requirements mapping to verify the methods to fulfill the requirements in NASA-STD-8739.8.
   

   Software requirements tailoring is the process used to seek relief from standard requirements consistent with program or project objectives, acceptable risk, and constraints. To accommodate the wide variety of software systems and subsystems, application of these requirements to specific software assurance, software safety, and IV&V efforts may be tailored where justified and approved.

   NASA established the Technical Authority governance model to approve and mitigate any changes to the application of the requirements in the Software Assurance and Software Safety Standard. Tailoring from requirements in the Software Assurance and Software Safety Standard are governed by the following requirements, as well as those established in NPR 8715.3 for all of the Agency’s investment areas. The responsible program, project, or operations manager must formally accept the risk. Tailoring at the Center level is to be decided by the SMA Technical Authority. The software assurance, software safety, and IV&V requirements are tailored using the following levels:

   1. 1. The first level of tailoring is the Software Classification Decision, see NPR 7150.2 ⁰⁸³.
      2. The second level of tailoring is the tailoring in the project’s Software Requirements Mapping Matrix, see NPR 7150.2.
      3. The third level of tailoring is the tailoring by the Software Assurance TA of the Software Assurance and Software Safety requirements that correspond to the project’s Software Requirements Mapping Matrix requirements.

   The Software Assurance and Software Safety Standard establishes a baseline set of requirements to reduce software assurance, software safety, and IV&V risks on NASA projects and programs. Each project has unique circumstances, and tailoring can be employed to modify the requirements set appropriate for software assurance, software safety, and IV&V effort.

   Determining the tailoring of requirements is a joint software engineering effort and SMA effort, including acceptable technical and programmatic risk posture, Agency priorities, size, and complexity. Requirements can be tailored more broadly across a group of similar projects, a program, an organization, or another collection of similar software development efforts.

   The software assurance organization should maintain a requirements mapping matrix or multiple requirements mapping matrices against requirements in the Software Assurance and Software Safety Standard, including those delegated to other parties or accomplished by contract vehicles.

   The software assurance organization will conduct risk assessment efforts to determine the software assurance, software safety, and IV&V tasks to be performed, and the rigor of each task. The software assurance organization will develop, maintain, and execute plans and procedures that cover the entire software life cycle and, as a minimum, address the requirements of the Software Assurance and Software Safety Standard with approved tailoring.

   The approval of the Software Assurance and Software Safety Standard tailoring is determined by the appropriate SMA management at the Center Level in conjunction with the project.

   The request for relief from a requirement includes:

   • • the rationale,
     • a risk evaluation, and
     • reference to all material that justifies supporting acceptance.

   The organization submitting the tailoring request informs the next higher level of involved management promptly of the tailoring request. The dispositioning organization reviews the request with the other organizations that could be impacted or have a potential risk (i.e., to safety, quality, cybersecurity, health) with the proposed changes; and obtains the concurrence of those organizations.

   The Center SMA technical authority shall review and agree with any tailored Software Assurance and Software Safety Standard requirements. (SASS-9)

   If a system or subsystem development evolves to meet a higher or lower software classification as defined in NPR 7150.2 then the software assurance, software safety, and IV&V organizations shall update their plan(s) to fulfill the applicable requirements per the Requirements Mapping Matrix and any approved changes, and initiate adjustments to applicable contracts to meet the modified requirements. (SASS-10)
   
   

2. Perform an audit of SA performance against the requirements in SA plans at least every two years or upon request - Plan and conduct an audit to verify that the selected software assurance and software safety requirements are being followed by whoever is responsible for performing the software assurance and software safety activities, the software assurance and software safety products exist, and the software assurance and software safety metrics are being recorded and used. 
   
   
3. Monitor issues, corrective actions, findings to closure. - identify any issues with meeting and executing the software assurance and safety standard requirements and with the implementation of NPR 7150.2 requirements, identify risks associated with not meeting any of the requirements, and track all issues to closure.
   
   

Periodic audits against SA plans and requirements as well as regular status monitoring and daily activities with the engineering teams can help identify requirements that are not being met and the associated risks.