2.2.10 The project shall implement software assurance per NASA-STD-8739.8, NASA Software Assurance Standard.
Software assurance activities occur throughout the life of the project. Engineering or the project may perform some of the actual analyses and activities. NASA's Safety and Mission Assurance organizations provide assurance that the products and processes are implemented according to the agreed upon plan(s). Software assurance is recommended on software activities and products, including solicitations, contract and memorandums of agreements, software plans, requirements, design, implementation, verification, validation, certification, acceptance, maintenance, operations, and retirement activities.
1.2 Applicability Across Classes
The team is to implement the software engineering products and processes according to the plans established by the project. Software assurance activities ensure this happens through a series of reviews and audits. For NASA, NASA-STD-8739.8 captures the assurance requirements in a single place for more uniform application across projects where software is developed or acquired for NASA. Topic 7.16 - Traceability of 7150.2 to other NPRs, NASA-STDs in this Handbook includes a table that shows the relationship between the software engineering practices described in the NPR 7150.2 requirements and the software assurance requirements documented in NASA-STD-8739.8. Understanding these relationships can help project personnel see requests from software assurance as reasonable and be more accepting of results from software assurance activities.
Software assurance activities begin during the initiation or pre-award phase of a project (before a Request for Proposal (RFP) is issued), continue through maintenance and retirement, and are performed by persons other than those carrying out the process or creating the product that is being assured.
Software assurance includes the disciplines of software quality, software safety, software reliability, software verification and validation (V&V), and independent V&V. All of these activities work together to ensure high quality products that operate safely. The amount of software assurance required for a project is based on the classification of the software SWE-020 and the safety criticality SWE-133 of the software.
NASA-STD-8739.8 defines roles, not individuals, so software assurance tasks can be carried out by whoever is assigned to fill the named role. It is preferable, but not necessary, for a dedicated software assurance group to carry out the software assurance activities. If such a group does not exist or is not available, the key is to have an organization, group, or person independent (managerially separate) from the organization creating the software and, per NASA-STD-8739.8, "either perform or assure that software assurance activities are performed correctly and to the necessary level, and that records of those activities are created, analyzed, and maintained." More information on the roles and independence required for software assurance can be found in NASA-STD-8739.8, which is also listed in the Resources section.
To avoid misunderstandings and issues once the project has begun, the following software assurance items need to be considered for inclusion in the provider's contract:
- Access to provider processes and documentation, see SWE-040.
- Software assurance activities, meetings, reviews, etc. planned for the project.
- Standards and requirements for provider-performed software assurance activities.
- Software assurance metrics to be collected and reported by the software provider.
Use checklists, documented procedures, templates, etc. to ensure consistency among software assurance activities across projects.
Consult Center Process Asset Libraries (PALs) for Center-specific guidance and resources related to software assurance.
4. Small Projects
Software assurance is required regardless of project size. Since NASA-STD-8739.8 defines roles, not individuals, projects with limited personnel resources, can use one person to fulfill multiple roles and perform multiple software assurance functions, as long as the proper independence for the specific requirement is maintained. In small project situations, sometimes it will be necessary for a project to have software assurance conducted by someone not on the project, but potentially from the same organization. This is not the preferred approach, but better than having no software assurance done at all.
Additionally, for acquired software, software assurance is performed by both the acquirer and the provider, so projects with small acquirer staffs could consider doing more "insight" (surveillance mode requiring the monitoring of customer-identified metrics and contracted milestones) per NASA-STD-8739.8, "Tailoring the implementation of software assurance requirements is acceptable commensurate with the program/project classification as well as size, complexity, criticality, and risk."
6. Lessons Learned
Software quality assurance implementation is a balancing activity that must be tailored as project appropriate.
"No project in the history of Software Development at NASA has had 'enough' money, especially when it comes to implementing Software Quality Assurance activities. It is not possible to achieve all aspects of quality because of interrelationships. Software Quality Assurance engineers must determine, based on experience, what trade-offs are to be made within the specific project since each project has different objectives. Gilles defined some of these relationships between the quality assurance criteria e.g., an inverse relationship between usability and efficiency; a direct relationship between maintainability and reusability 2. To make the best trade-offs, all relationships must be understood and experience used to interpret the impact." (Lesson 2 of 12)
Software quality assurance (SQA) must evaluate the process as well as the products.
"Within NASA, SQA tends to focus on the products produced, such as the requirements documents, design, code listing, and test plans. But the focus of SQA is to monitor continuously throughout the Software Development Life Cycle to ensure the quality of the delivered products. This requires monitoring both the processes and the products. In process assurance, SQA provides management with objective feedback regarding compliance to approved plans, procedures, standards, and analyses. Product assurance activities focus on the changing level of product quality within each phase of the life cycle, such as the requirements, design, code, and test plan. The objective is to identify and eliminate defects throughout the life cycle as early as possible, thus reducing test and maintenance costs." (Lesson 3 of 12)
There must be a software assurance plan.
"Most project managers feel they have too many plans, and the suggestion of another one for SQA might be the proverbial straw that breaks the camel's back! But the success of any undertaking is to know what one is trying to achieve and how one expects to accomplish it, hence, the necessity of a plan for SQA. The plan will specify the goals, what is to be performed, standards against which the development work is to be measured, all relevant procedures, and the organizational structure of the Quality Assurance within the project. At NASA a software assurance plan is required." (Lesson 4 of 12)
Software quality assurance must span the entire software development life cycle.
"Software Development starts at the concept phase and continues through maintenance. SQA activities and funding should also start at the concept definition and continue through the entire life cycle. At each phase, there are activities that could be performed 4. The project and the quality assurance office must work together to negotiate what activities are appropriate at each phase based on risks and resources. The Quality Assurance Plan should indicate what activities will be performed, the decision basis, and the trade-offs made." (Lesson 5 of 12)
Requirements are the birthplace of successful projects.
"Although SQA is performed across the entire life cycle, success of a project can often be determined by the attention paid to requirements. The earlier in the life cycle potential risks are identified, the easier it is to eliminate or at least manage the conditions that introduce that risk. Problems not found until the testing phase are approximately 14 times more costly to fix than if found and fixed earlier in the requirements phase. The first tangible representation of the capabilities produced is the requirements specification document, be they system, hardware, software, or operational requirements. The document also serves to establish the basis for all the project's engineering management and assurance functions. If the quality of the requirements specification is poor, the project is at risk even before work begins.7 Therefore, a specific lesson in SQA is the emphasis on requirements." (Lesson 6 of 12)
Software quality assurance does not equal testing.
"Too often project managers assume they have Quality Assurance since they are doing testing already, or believe they don't need quality assurance until testing. These are incorrect assumptions based on lack of experience, understanding and/or knowledge. From the aspect of quality assurance, the purpose of testing is to:
- Assure problems are documented, corrected, and used for process improvement.
- Assure problem reports are assessed for their validity.
- Assure reported problems and their associated corrective actions are implemented in accordance with customer approved solutions.
- Provide feedback to the developer and the user of problem status.
- Provide data for measuring and predicting Software Quality and Software Reliability."
(Lesson 7 of 12)
Metrics are a necessity.
"Software Metrics are often ignored during the early Software Development Life Cycle phases and not actively associated with SQA - but should be. For SQA practitioners, with their responsibility for assuring both the processes and the products of the Software Development, measurement is critical. Throughout each of the life cycle phases metrics have proven invaluable in the evaluation of the quality of the products and processes 5." (Lesson 8 of 12)
Safety and reliability are critical aspects of SQA.
"Safety is a team effort and everyone's responsibility. Software within NASA is a vital part of the system and can impact the safety of the mission. Project managers, Systems Engineers, Software Leads and engineers, Software Assurance or Quality Assurance, and System Safety Personnel all play a part in creating a safe system. Reliability and safety cannot be tested at the end of a project; they must be built in as the software is being Developed.
Reliability impacts safety - a system cannot be deemed safe if it is not reliable." (Lesson 9 of 12)
Independent verification and validation (IV&V) is an important tool within SQA.
"NASA defines Software IV&V as a Systems Engineering Process employing rigorous methodologies for evaluating the correctness and quality of the software product throughout the Software Development Life Cycle. Without SQA, IV&V is expensive and often not as effective. SQA is a broad 'blanket' across the project, overseeing all process and product activities, including software. IV&V focuses on only those processes and products determined to have the highest risk, doing an in-depth evaluation of them. IV&V should not be used on all projects, but instead as a tool to increase reliability and the probability of mission success! 4" (Lesson 10 of 12)
Hardware does not equal software!
"The influence of Hardware Quality Assurance is evident in the Software Quality Assurance practitioner community, where hardware-intensive systems and typical hardware-related concerns predominate. Two issues which dominate discussions about hardware are time and operating conditions. Software, however, is built with different constraints and considerations. NASA has grappled with these differences and the best approach for recognizing the differences (yet similarities) of hardware and software quality assurance. At Goddard Space Flight Center (GSFC), and within other NASA Centers, hardware and software QA are in one department. This allows for them to work together, since both are needed for a successful project, yet are still separate in other areas where needed." (Lesson 11 of 12)
Risk Management is not optional.
"Risk is a daily reality on all projects, and Continuous Risk Management should become just as routine. It should be ongoing and comfortable and neither imposed nor forgotten. Like any good habit, it should seamlessly fit into the daily work.
Software Risk Management is important because it helps avoid disasters, rework, and overkill. The objectives of Software Risk Management are to identify, address, and eliminate software risk items before they become threats to success or major sources of rework. Good project managers are also good managers of risk. It makes good business sense for all software development projects to incorporate risk management as part of project management." (Lesson 12 of 12)