1.  Introduction

A main software assurance objective is to provide a level of confidence that the software produced is error free and performs in its intended manner. Thus, the software assurance personnel are involved in projects throughout the entire software development life cycle monitoring the development activities, ensuring that the correct standards, processes, and procedures are being followed, and evaluating the product quality as it is being developed. Audits are a common tool used to assess the quality of the software products and the level of compliance with the process requirements.

The Audit Reports topic focuses on the many aspects of software auditing and how to report the results. Since audits and auditing are discussed in many areas of this Handbook, some audit topics will provide links to other areas of the Handbook where the information is already located. For example, Topic 8.12 - Basics of Software Auditing contains an extensive description of the basic process of planning and conducting an audit, along with reporting the results. 

The information in this topic is divided into several tabs as follows:

• Tab 1 – Introduction.
• Tab 2 – Audit requirements and recommendations in the Software Assurance and Software Safety Standard (NASA-STD-8739.8).
• Tab 3 – Audit Metrics that are typically collected or associated with audits and their use.
• Tab 4 – Audit Schedules discusses when to conduct process/procedure audits
• Tab 5 – Audit and Assessment Checklists provides a list of the available Process Asset Templates (PATs) that may be used for process audits and SWE work product assessments.
• Tab 6 – Small Project Audits gives tips to small projects to minimize cost
• Tab 7 – Audit Report Contents discusses the communication of the audit results.
• Tab 8 – Resources for this topic.

1.1 Additional Guidance

Links to Additional Guidance materials for this subject have been compiled in the Relevant Links table. Click here to see the Additional Guidance in the Resources tab.

See also SWE-022 - Software Assurance for Audit Reports listed in SA Work Products. 

2. Audits

Auditing is one tool used by the software assurance and software safety personnel to determine whether projects are following and in compliance with governing requirements and standards along with project processes and procedures. Thus, the Software Assurance and Software Safety (SASS) Standard (NASA-STD-8739.8) lists requirements for many types of audits and required tasks that could be performed by auditing (e.g., compliance assessments).

When conducting audits, there are two basic types – Formal and Informal. Formal audits are discussed in Topic 8.12 - Basics of Software Auditing. Informal have less rigor and more leeway on how they are conducted.

• Formal Audits – Formal audits involve carefully selected areas, outcomes, and samples. Some characteristics are:
  • Audit team typically includes members from external organizations
  • Audits focus on comprehensive process/procedure audits, typically reviewing the organization
  • Results are reported to upper management or corporate executives
  • Audits are against organizational or industry standards
  • Findings may result in corporate repercussions such as the loss of a certification or contract
  • Audits use formal/structured reporting and documentation
  • Audit Planning is formal and structured; coordination and communication are very formal and well documented
• Informal Audits – Informal audits are generally much smaller and may be performed on an ad hoc basis. Some characteristics are:
  • Audit team members are primarily from internal teams
  • Audits often check one process/procedure at a time, typically looking at the project
  • Results are reported to project management
  • Audits are against project processes/ procedures/ standards
  • Findings may be resolved by the project
  • Reporting and documentation are more informal (e.g., SA Status Reports)
  • Audit Planning is coordinated and communicated informally

There are three basic audit perspectives – targeted, generic, and those that satisfy other requirements. The targeted audits focus on one specific process, product, or activity (e.g., software configuration management audits). Generic audits are more general in nature and may call for audits of standards, processes, procedures, and practices. Those audits may require multiple audits, spanning the life cycle to ensure all aspects are met (e.g., software development audits). A third perspective of audits is comprised of those requirements in the SASS Standard listed as “confirm” or “assess” that can be easily satisfied by performing an audit (e.g., SASS tasking for SWE-139 which calls for assessing compliance to NPR 7150.2). All three audit perspectives should be considered when planning the schedule of audits for a project. All project audits should be planned in conjunction with the activities that are occurring in that time period and need to be coordinated with the project schedule. For example, it does not make sense to plan an audit to evaluate the test procedures during the planning phase when they have not been developed yet.

As mentioned previously in the Introduction, there are many good resources on planning, conducting, and closing out an audit in Topic 8.12 - Basics of Software Auditing.  As part of the planning for an audit, checklists should be generated or used. To make this easier, there are many checklists available in this Handbook as well as in Center asset libraries, which can be accessed from the NASA Engineering Network (NEN) Software Community of Practice (CoP) Software Processes Across NASA (SPAN) ¹⁹⁷ site. Once on the Software Engineering CoP, select the “Process Assets (SPAN)” option to access the NASA SPAN ¹⁹⁷ site. A list of audit checklists can be found in this Handbook in Topics 5.03 - Inspect - Software Inspection, Peer Reviews, Inspections and 8.17 - Software Safety Audit Checklists. As part of the audit preparation, checklists should be reviewed as they may need to be modified to fit specific tailored, approved processes that are in place for the project.

After the audit is conducted, a key part of any audit involves the post audit activities and actions. Any type of non-conformance, including findings, issues, defects, or errors found during an audit is recorded in a tracking system and these non-conformances are tracked to closure by the audit team. An analysis or assessment of the non-conformances should be performed and compared with previous or similar audit results to determine if there are systemic or consistent problems, which might require a closer look.

Most of the audits that Software Assurance will perform are of the informal nature and targeted to software particular process, procedure, activity, or product. Table 1 below lists the required Software Assurance and Software Safety Standard (SASS) audits along with the associated SWE requirement number. Table 2 lists some of the SASS Tasks where an audit might be one way to satisfy the requirement but is not the only method that could be used.

Table 1: Audit Requirements in the Software Assurance and Software Safety Standard (NASA-STD-8739.8 ²⁷⁸):

Table 2: Requirements in NASA-STD-8739.8 that might be satisfied with the use of an audit:

2.1 Additional Guidance

Links to Additional Guidance materials for this subject have been compiled in the Relevant Links table. Click here to see the Additional Guidance in the Resources tab.

3. Audit Metrics

Audit metrics may be collected and reported on at the organization or project-level. Those collected at the organization-level may be the result of organization-level audits or the roll-up of the audit data from individual projects to see how the organization is performing as a whole. The audit metrics collected at the project-level are specific to the project and should be able to provide insight into the health and status of the project.

The audit metrics that are tracked and reported on will depend on the specific metrics chosen by the audit team. The Audit Team may be from a project, an independent 3rd party auditors for commercial ventures, or organization. The audit team must select or develop the audit metrics that best reflect the goals of audit to be performed. Organizations or projects may have a pre-defined set of metrics making the selection process straight forward. Typically, only a few audit metrics are selected, as the most important things to come out of an audit are the findings and observations themselves.

Topic 8.18 - SA Suggested Metrics provides a list of potential metrics including audit metrics that could be selected for specific requirements and are sorted by the type of metrics they support (e.g., Peer Review Process Audits, Compliance Audits). This is not an exhaustive list and may not cover all of the possible audit metrics that need to be collected and reported.

When selecting metrics, the audit team may want to look at the data from different aspects. Selecting a few high-level metrics will provide overall status information. Some of these should be chosen to give the management team a quick view of the “state of the project,” or the “state of the Software Assurance work” being done. A couple of examples high-level audit metrics are:  

• # of audits performed vs. # of audits planned 
• # of Open vs. Closed Audit Non-Conformances over time

Selecting a few low-level metrics will attest to the quality of the work that was performed. The lower-level metrics should be chosen to give the management team insight into how well the projects and organizations are following the NASA requirements and standards as well as their organizational and project processes and procedures. A couple of examples of low-level requirements are:  

• # of Non-Conformances identified in the CM Plan
• # of audit Non-Conformances per peer review audit

Some metrics examples that could be chosen:

3.1 Audits of Software Assurance:

• # of audits performed versus # of audits planned
• # of Peer Review Audits planned vs. # of Peer Review Audits performed
• # of Compliance Audits planned vs. # of Compliance Audits performed
• Trends on non-conformances from audits (Open, Closed, life cycle Phase)
• Time required to close audit Non-Conformances
• Preparation time each audit participant spent preparing for audit

3.2 Peer Review Process Audit Metrics:

• # of audit Non-Conformances per Peer Review Audit
• # of audit Observations per Peer Review Audit
• # of Peer Review Audits planned vs. # of Peer Review Audits performed
• Trends on non-conformances from audits (Open, Closed, Life cycle Phase)
• Time required to close Peer Review Audit Non-Conformances
• Preparation time each audit participant spent preparing for audit

3.3 Compliance Audit Metrics:

• # of Compliance Audits planned vs. # of Compliance Audits performed
• # of Open vs. Closed Audit Non-Conformances over time
• Trends of # of Non-Conformances from audits over time (Include counts from process and standards audits and work product audits.)
• # of Non-Conformances identified in plans (e.g., SMPs, SDPs, CM Plans, SA Plans, Safety Plans, Test Plans)
• # of Non-Conformances identified in the software Configuration Management Plan
  • Trends of # Open vs. # Closed over time
• # of Configuration Management Audits conducted by the project – Planned vs. Actual
• # of Non-Conformances per audit (including findings from process and compliance audits, process maturity)
• # of process Non-Conformances (e.g., activities not performed) identified vs. # accepted by the project
  • Trends of # Open vs. # Closed over time"
• # of process Non-Conformances in the Risk Management process identified

3.4 Project health and status based on Audit metrics:

• # of process Non-Conformances (e.g., activities not performed) identified vs. # accepted by the project
• # of Compliance Audits planned vs. # of Compliance Audits performed
• # of Open vs. Closed Audit Non-Conformances over time
• Trends of # of Non-Conformances from audits over time (Include counts from process and standards audits and work product audits.)

3.5 SA value in Peer Reviews:

• How many non-conformances have been found in each peer review? # of non-conformances found in each peer review
• # of non-conformances found by SA
• # of non-conformances accepted by the project
• Trend of open non-conformances vs closed non-conformances over time (SA analysis of trends could show project problems if # of closed non-conformances continue to lag further and further behind # of open non-conformances

3.6 Observations of Project status based on Audit metrics:

• # of process Non-Conformances (e.g., activities not performed) identified by SA vs. # accepted by the project
• # of Compliance Audits planned vs. # of Compliance Audits performed
• # of Open vs. Closed Audit Non-Conformances over time
• Trends of # of Non-Conformances from audits over time (Include counts from process and standards audits and work product audits.)

3.7 Safety issues and non-conformances:

• # of safety related-requirements issues (over time, open, closed)
• # of safety-related non-conformances identified by life cycle phase (over time, open, closed)

3.8 Display Data With Charts

Simple charts are a good way to show the data for discussion when reporting status. Some simple examples are below:

3.9  Additional Guidance

Links to Additional Guidance materials for this subject have been compiled in the Relevant Links table. Click here to see the Additional Guidance in the Resources tab.

4. Audit and Assessment Schedules

Software Assurance is required to perform audits on software development and software assurances processes along with performing assessments on the software work products. A schedule establishes when audits and assessments will be performed within a specified timeframe for a single project, all projects, or a program. The software development processes and practices are required to be audited at least once every two years, which means auditing a project to ensure that the project follows defined processes and procedures for deliverable software products. Assessments should be performed on each work product that the development team produces. Audits and assessments may be triggered by project milestones, changes in governing documents (e.g., a new version of an applicable standard is released, and the project has a specified timeframe by which to comply), major staffing changes, or identification of a major defect. Auditors may choose to conduct the audits all at once or throughout the life cycle as it is being executed. For example, create a schedule to audit all areas of a project’s software development plan over a year so that at the end of the year, all areas of the software development plan have been audited. It is recommended that the work product assessments be performed as the work products are being completed or baselined.

The tables below provides the recommended schedule for Software Assurance to use when conducting the software engineering process/procedure audits and work product assessments prescribed in NPR 7150.2 and NASA-STD-8739.8. The green blocks represent the recommended milestone where it is optimal to conduct the audit or assessment. If the audit or assessment cannot be conducted during the recommended time frame, the yellow blocks indicate a good secondary time frame.

4.1 Software Assurance Process Audits Schedule

4.2 Software Assurance Process Assessment Schedule

4.3 Minimum Required Audits

There are 4 process/procedure area audits that must be performed:

1. Software Configuration Management – This includes:
   1. Performing audits for all deliveries per the configuration management processes to verify that all products are being delivered and are the correct versions.
   2. Performing an audit against the configuration management procedures to confirm that the project follows the established procedures.
   3. Confirming that the project manager performed software configuration audits to determine the correct version of the software configuration items and verify that the results of the audit conform to the records that define them.
2. Software Requirements
3. Software Testing
4. Software Defects Tracking

Software Engineering Work Product audits should be conducted in-phase with the appropriate milestone review.

All other audits are at the discretion of the project but should be performed. The project’s Requirements Mapping Matrix should reflect which audits are being tailored.

5. Audit & Assessment Checklists

5.1 Audit Checklists for SWE Requirement Compliance  

5.2 Assessment Checklists for SWE Work Products

5.3 Other Audit and Assessment Checklists

-- Under construction

6. Small Project Audits

Projects with small budgets or limited personnel may need to strategically optimize the types and number of audits conducted. Here are some potential solutions on how to do that:

• Limit the number of process audits. At a minimum, perform the minimum require audits that are prescribed by this topic (See the Audit Schedules on Tab 4 - Audit Schedules)
• Instead of maintaining a separate tracking system/database for audit findings, enter the findings in the project’s defect tracking system.
  • The tracking system could be the project’s defect database or as simple as adding an inline comment to a Confluence page that logs change traffic.
• Perform informal audits versus the more rigorous formal audits.
• Audit team may be comprised of one person, but that person should not be the person who performed the activities being audited.

7. Audit or Assessment Report Content

8. Resources

8.1 References

8.2 Tools 

8.3 Additional Guidance

Additional guidance related to this requirement may be found in the following materials in this Handbook:

8.4 Center Process Asset Libraries

See the following link(s) in SPAN for process assets from contributing Centers (NASA Only).