1. Requirements

3.6.1 The project manager shall plan and implement software assurance per NASA-STD-8739.8.

1.1 Notes

Software assurance activities occur throughout the life of the project.  Some of the actual analyses and activities may be performed by engineering or the project. 

1.2 Applicability Across Classes

Key:    - Applicable | - Not Applicable

2. Rationale

“The purpose of software assurance is to assure that software products are of sufficiently high quality and operate safely and reliably... The software assurance process is the planned and systematic set of activities that ensure conformance of software life cycle processes and products to requirements, standards, and procedures.  Software assurance assures that the software and its related products meet their specified requirements, conform to standards and regulations, are consistent, complete, correct, safe, secure and reliable as warranted for the system and operating environment, and satisfy customer needs." ²⁷⁸

3. Guidance

“The purpose of software assurance is to assure that software products are of sufficiently high quality and operate safely and reliably.  This includes products delivered to and used within NASA, and products developed and acquired by NASA.”  ²⁷⁸ Software assurance assists in risk mitigation by helping expose potential defects in products and processes, thus preventing problems from evolving.  However, it also, through its metrics, tracking and analyses activities, enables improvement of future products and services.  Software assurance often serves as the corporate memory from project to project, sharing potential problem areas and lessons learned.  

The purpose of ... [the NASA Software Assurance] Standard is to:

• Establish a common framework consisting of activities and tasks that meet the objectives of software assurance in support of all software development processes.
• Establish an approach to planning to define and document the approach to performing software assurance on a given project to include the allocation of assurance tasks to specific groups and the intercommunication between those groups.
• Standardize the use of the independent reporting structure required for NASA safety, reliability and quality processes. ⁴⁶⁸ 

The outcomes of using this standard are:

1. A strategy for conducting software assurance is developed to include each of the software assurance disciplines
2. Evidence of software assurance activities are produced and maintained
3. Problems and/or non-conformances are identified and recorded; and
4. Adherence of products, processes and activities to the applicable standards, procedures and requirements are confirmed ⁴⁶⁸ 

At NASA, software assurance has evolved into an umbrella risk identification and mitigation strategy for safety and mission assurance of all NASA’s software. “It provides a consistent, uniform basis for defining the requirements for software assurance programs to be applied and maintained throughout the life of that software, that is, from project conception, through acquisition, development, operations and maintenance, and then evaluates if the software is properly retired.” ²⁷⁸  Software assurance activities begin during the initiation or pre-award phase of a project (before a Request for Proposal (RFP) is issued), continue through maintenance and retirement, are planned at the project level, and performed by persons other than those carrying out the process or creating the product that is being assured.

Software engineering and the software assurance disciplines are integrally related and yet each has its own responsibilities.  Jointly they are responsible for providing project management with the optimal solution for software to meet the engineering, safety, quality, and reliability needs of the project.  This necessitates a close working relationship to establish the appropriate levels of effort for both.  The NASA Procedural Requirements, NPR 7150.2, NASA Software Requirements ⁰³⁹ invokes the NASA Software Assurance Standard (NASA-STD-8739.8) ²⁷⁸and the NASA Software Safety Standard (NASA-STD-8719.13) ²⁷¹, requiring a close working relationship, understanding of roles and responsibilities, and establishing expected communication paths.   NPR 7150.2, besides laying out the NASA minimum requirements for software development, provides the NASA software classification upon which software engineering, software assurance and software safety all base their tailoring. ⁴⁶⁸  

Software assurance includes the disciplines of software quality, software safety, software reliability, software verification and validation (V&V), and independent V&V. All of these activities work together to ensure high quality products that operate safely. The amount of software assurance required for a project is based on the classification of the software SWE-020 and the safety criticality SWE-133 of the software.

NASA's Safety and Mission Assurance organizations provide assurance that the products and processes are implemented according to the agreed upon plan(s). Software assurance is recommended on software activities and products, including solicitations, contract and memorandums of agreements, software plans, requirements, design, implementation, verification, validation, certification, acceptance, maintenance, operations, and retirement activities.

NASA-STD-8739.8 defines roles, not individuals, so software assurance tasks can be carried out by whoever is assigned to fill the named role. It is preferable, but not necessary, for a dedicated software assurance group to carry out the software assurance activities. If such a group does not exist or is not available, the key is to have an organization, group, or person independent (managerially separate) from the organization creating the software and, per NASA-STD-8739.8, "either perform or assure that software assurance activities are performed correctly and to the necessary level, and that records of those activities are created, analyzed, and maintained." More information on the roles and independence required for software assurance can be found in NASA-STD-8739.8, ²⁷⁸ which is also listed in the Resources section.

To avoid misunderstandings and issues once the project has begun, the following software assurance items need to be considered for inclusion in the provider's contract (Topic 7.03 - Acquisition Guidance):

• Access to provider processes and documentation, see SWE-040.
• Software assurance activities, meetings, reviews, etc. planned for the project.
• Standards and requirements for provider-performed software assurance activities.
• Software assurance metrics to be collected and reported by the software provider.

Use checklists, documented procedures, templates, etc. to ensure consistency among software assurance activities across projects.

The team is to implement the software engineering products and processes according to the plans established by the project. Software assurance activities ensure this happens through a series of reviews and audits. For NASA, NASA-STD-8739.8 captures the assurance requirements in a single place for more uniform application across projects where software is developed or acquired for NASA.

NASA-specific software assurance resources are available in Software Processes Across NASA (SPAN), accessible to NASA users from the SPAN tab in this Handbook. 

4. Small Projects

Software assurance is required regardless of project size. Since NASA-STD-8739.8 defines roles, not individuals, projects with limited personnel resources, can use one person to fulfill multiple roles and perform multiple software assurance functions, as long as the proper independence for the specific requirement is maintained. In small project situations, sometimes it will be necessary for a project to have software assurance conducted by someone not on the project, but potentially from the same organization. This is not the preferred approach, but better than having no software assurance done at all.

Additionally, for acquired software, software assurance is performed by both the acquirer and the provider, so projects with small acquirer staffs could consider doing more "insight" (Surveillance mode requiring the monitoring of customer identified metrics and contracted milestones) per NASA-STD-8739.8 ²⁷⁸, "Tailoring the implementation of software assurance requirements is acceptable commensurate with the program/project classification as well as size, complexity, criticality, and risk."

5. Resources

6. Lessons Learned

• Software quality assurance implementation is a balancing activity that must be tailored as project appropriate.

"No project in the history of Software Development at NASA has had 'enough' money, especially when it comes to implementing Software Quality Assurance activities. It is not possible to achieve all aspects of quality because of interrelationships. Software Quality Assurance engineers must determine, based on experience, what trade-offs are to be made within the specific project since each project has different objectives. Gilles defined some of these relationships between the quality assurance criteria e.g., an inverse relationship between usability and efficiency; a direct relationship between maintainability and reusability ². To make the best trade-offs, all relationships must be understood and experience used to interpret the impact." (Lesson 2 of 12) ⁴⁴⁴

• Software quality assurance (SQA) must evaluate the process as well as the products.

"Within NASA, SQA tends to focus on the products produced, such as the requirements documents, design, code listing, and test plans. But the focus of SQA is to monitor continuously throughout the Software Development Life Cycle to ensure the quality of the delivered products. This requires monitoring both the processes and the products. In process assurance, SQA provides management with objective feedback regarding compliance to approved plans, procedures, standards, and analyses. Product assurance activities focus on the changing level of product quality within each phase of the life cycle, such as the requirements, design, code, and test plan. The objective is to identify and eliminate defects throughout the life cycle as early as possible, thus reducing test and maintenance costs." (Lesson 3 of 12) ⁴⁴⁴

• There must be a software assurance plan.

"Most project managers feel they have too many plans, and the suggestion of another one for SQA might be the proverbial straw that breaks the camel's back! But the success of any undertaking is to know what one is trying to achieve and how one expects to accomplish it, hence, the necessity of a plan for SQA. The plan will specify the goals, what is to be performed, standards against which the development work is to be measured, all relevant procedures, and the organizational structure of the Quality Assurance within the project. At NASA a software assurance plan is required." (Lesson 4 of 12) ⁴⁴⁴

• Software quality assurance must span the entire software development life cycle.

"Software Development starts at the concept phase and continues through maintenance. SQA activities and funding should also start at the concept definition and continue through the entire life cycle. At each phase, there are activities that could be performed ⁴. The project and the quality assurance office must work together to negotiate what activities are appropriate at each phase based on risks and resources. The Quality Assurance Plan should indicate what activities will be performed, the decision basis, and the trade-offs made." (Lesson 5 of 12) ⁴⁴⁴

• Requirements are the birthplace of successful projects.

"Although SQA is performed across the entire life cycle, success of a project can often be determined by the attention paid to requirements. The earlier in the life cycle potential risks are identified, the easier it is to eliminate or at least manage the conditions that introduce that risk. Problems not found until the testing phase are approximately 14 times more costly to fix than if found and fixed earlier in the requirements phase. The first tangible representation of the capabilities produced is the requirements specification document, be they system, hardware, software, or operational requirements. The document also serves to establish the basis for all the project's engineering management and assurance functions. If the quality of the requirements specification is poor, the project is at risk even before work begins.⁷ Therefore, a specific lesson in SQA is the emphasis on requirements." (Lesson 6 of 12) ⁴⁴⁴

• Software quality assurance does not equal testing.

"Too often project managers assume they have Quality Assurance since they are doing testing already, or believe they don't need quality assurance until testing. These are incorrect assumptions based on lack of experience, understanding and/or knowledge. From the aspect of quality assurance, the purpose of testing is to:

• Assure problems are documented, corrected, and used for process improvement.
• Assure problem reports are assessed for their validity.
• Assure reported problems and their associated corrective actions are implemented in accordance with customer approved solutions.
• Provide feedback to the developer and the user of problem status.
• Provide data for measuring and predicting Software Quality and Software Reliability."

(Lesson 7 of 12) ⁴⁴⁴

• Metrics are a necessity.

"Software Metrics are often ignored during the early Software Development Life Cycle phases and not actively associated with SQA - but should be. For SQA practitioners, with their responsibility for assuring both the processes and the products of the Software Development, measurement is critical. Throughout each of the life cycle phases metrics have proven invaluable in the evaluation of the quality of the products and processes ⁵." (Lesson 8 of 12) ⁴⁴⁴

• Safety and reliability are critical aspects of SQA.

"Safety is a team effort and everyone's responsibility. Software within NASA is a vital part of the system and can impact the safety of the mission. Project managers, Systems Engineers, Software Leads and engineers, Software Assurance or Quality Assurance, and System Safety Personnel all play a part in creating a safe system. Reliability and safety cannot be tested at the end of a project; they must be built in as the software is being Developed.
Reliability impacts safety - a system cannot be deemed safe if it is not reliable." (Lesson 9 of 12) ⁴⁴⁴

• Independent verification and validation (IV&V) is an important tool within SQA.

"NASA defines Software IV&V as a Systems Engineering Process employing rigorous methodologies for evaluating the correctness and quality of the software product throughout the Software Development Life Cycle. Without SQA, IV&V is expensive and often not as effective. SQA is a broad 'blanket' across the project, overseeing all process and product activities, including software. IV&V focuses on only those processes and products determined to have the highest risk, doing an in-depth evaluation of them. IV&V should not be used on all projects, but instead as a tool to increase reliability and the probability of mission success! ⁴" (Lesson 10 of 12) ⁴⁴⁴

• Hardware does not equal software!

"The influence of Hardware Quality Assurance is evident in the Software Quality Assurance practitioner community, where hardware-intensive systems and typical hardware-related concerns predominate. Two issues which dominate discussions about hardware are time and operating conditions. Software, however, is built with different constraints and considerations. NASA has grappled with these differences and the best approach for recognizing the differences (yet similarities) of hardware and software quality assurance. At Goddard Space Flight Center (GSFC), and within other NASA Centers, hardware and software QA are in one department. This allows for them to work together, since both are needed for a successful project, yet are still separate in other areas where needed." (Lesson 11 of 12) ⁴⁴⁴

• Risk Management is not optional.

"Risk is a daily reality on all projects, and Continuous Risk Management should become just as routine. It should be ongoing and comfortable and neither imposed nor forgotten. Like any good habit, it should seamlessly fit into the daily work.

Software Risk Management is important because it helps avoid disasters, rework, and overkill. The objectives of Software Risk Management are to identify, address, and eliminate software risk items before they become threats to success or major sources of rework. Good project managers are also good managers of risk. It makes good business sense for all software development projects to incorporate risk management as part of project management." (Lesson 12 of 12) ⁴⁴⁴