1. Requirements

5.1.7 The project manager shall perform software configuration audits to determine the correct version of the software configuration items and verify that they conform to the records that define them.

1.1 Notes

NPR 7150.2, NASA Software Engineering Requirements, does not include any notes for this requirement.

1.2 History

1.3 Applicability Across Classes

Key:    - Applicable | - Not Applicable

2. Rationale

Configuration audits provide checks to ensure that the planned product is the developed product.

3. Guidance

3.1 Configuration Audits

For software configuration, audits help ensure that configuration items (CIs) have been developed and completed following the documents and requirements that define them. Audits also help ensure that CIs achieve their performance and functional characteristics goals and that all associated operational and support documents are complete and meet their requirements. Audits also determine if all CIs that are supposed to be part of the baseline or release is actually in the baseline or release and are the correct version and revision.

There are two types of configuration audits: the Functional Configuration Audit (FCA) and the Physical Configuration Audit (PCA). Configuration audits are performed for all releases; however, audits of interim, internal releases may be less formal and rigorous, as defined by the project.

3.2 Planning For Audits

Audit plans, including goals, schedules, participants, contractor participation, and procedures are documented in the configuration management (CM) plan (see 5.06 - SCMP - Software Configuration Management Plan).

When planning audits, it is important to remember that audits are samplings, not a look at every record.  It is also important to remember that auditors should not have any direct responsibility for the software products they audit.

The basic steps in an audit are:

The Department of Defense Configuration Management Guidance Handbook ³⁵¹ includes tables of activities for audit planning, preparation, performance, and close-out (generating the audit report and addressing corrective actions). These tables address both the Government and contractor roles in these activities and can be tailored as applicable for a project.

3.3 Functional Configuration Audit (FCA)

The SMA (Safety and Mission Assurance) Technical Excellence Program (STEP) Level 2 Software Configuration Management and Data Management course taught by the Westfall Team ³⁴³ suggests that the following be included in an FCA:

• "An audit of the formal test documentation against test data.
• "An audit of the verification and validation (V&V) reports.
• "A review of all approved changes.
• "A review of updates to previously delivered documents.
• "A sampling of design review outputs.
• "A comparison of code with documented requirements.
• "A review to ensure all testing was accomplished." ³⁴³
• Additional sample testing or rerunning of tests, as appropriate for the project.

A Checklist template for FCA can be found in PAT-001 - FCA Checklist. 

3.4 Physical Configuration Audit (PCA)

The STEP 2 course suggests that the following be included in a PCA ³⁴³:

• "An audit of the system specification for completeness [and removal of all to-be-determined (TBD)].
• "An audit of the FCA report for discrepancies & actions taken.
• "A comparison of the architectural design with the detailed design components for consistency.
• "A review of the module listing for compliance with the approved coding standards.
• "An audit of the manuals for format completeness & conformance to system & functional descriptions." ³⁴³

Additional audit topics to consider include:

• As coded, software products reflect their design.
• User documentation complies with standards as specified.
• Activities have been conducted according to applicable requirements, plans, and contracts.

A Checklist template for PCA can be found in PAT-002 - PCA Checklist. 

3.5 Data Typically Reviewed

NASA/SP-2007-6105, NASA Systems Engineering Handbook, ²⁷³ includes the following table showing the data typically reviewed during each of these audits:

3.6 When Should Audits Be Done?

Consider the following options when deciding when to conduct audits:

• At the time a product is released.
• Before delivery to assure that all delivered products are complete, contain the proper versions and revisions and that all discrepancies, openwork, and deviations, and waivers are properly documented and approved; can be FCA or PCA.
• At the end of a life cycle phase per Capability Maturity Model Integration (CMMI).
• Before the release of a new or revised baseline.
• As the project progresses to prevent finding major issues at the end when it's more costly to fix them and to identify systemic issues. such as meeting coding standards that could affect large segments of the project.
• Incrementally for very large, complex systems focusing on specific functional areas with a summary audit held to address the status of all identified action items (FCA). See also SWE-079 - Develop CM Plan, SWE-083 - Status Accounting.

See also SWE-085 - Release Management, 

3.7 Additional Guidance

Additional guidance related to this requirement may be found in the following materials in this Handbook:

3.8 Center Process Asset Libraries

See the following link(s) in SPAN for process assets from contributing Centers (NASA Only). 

4. Small Projects

For projects with limited personnel, consider sharing lead auditors or audit team members among projects. Another suggestion is for members of small projects to conduct configuration audits of other small projects.

5. Resources

5.1 References

5.2 Tools

5.3 Process Asset Templates

6. Lessons Learned

6.1 NASA Lessons Learned

A documented lesson from the NASA Lessons Learned database notes the following:

• Mars Climate Orbiter Mishap Investigation Board - Phase I Report. Lesson Number 0641 ⁵¹³: Configuration audits are called out as one of several mitigations for the root cause of the Mars Climate Orbiter (MCO) mishap.  One of the Recommendations states to "Conduct [a] software audit for specification compliance on all data transferred between JPL and [contractor]."

6.2 Other Lessons Learned

No other Lessons Learned have currently been identified for this requirement.

7. Software Assurance

7.1 Tasking for Software Assurance

7.2 Software Assurance Products

• None at this time

7.3 Metrics

• #  of Configuration Management Audits conducted by the project – Planned vs. Actual.
• # of Compliance Audits planned vs. # of Compliance Audits performed.
• # of software work product Non-Conformances identified by life cycle phase over time.
• # of Non-Conformances identified in release documentation (Open, Closed).
• # of process Non-Conformances (e.g., activities not performed) identified by SA vs. # accepted by the project.
• Trends of # Open vs. # Closed over time.
• # of Non-Conformances per audit (including findings from process and compliance audits, process maturity).
• # of Open vs. Closed Audit Non-Conformances over time.'
• Trends of # of Non-Conformances from audits over time (Include counts from process and standards audits and work product audits.).

See also Topic 8.18 - SA Suggested Metrics. 

7.4 Guidance

Software assurance should perform or participate in the project’s configuration audits, typically the Functional Configuration Audit (FCA) and the Physical Configuration Audit (PCA). The Functional Configuration Audit. A Functional Configuration Audit (FCA) examines the functional characteristics of the configured product and verifies that the product has met the requirements specified in its Functional Baseline documentation approved at the Preliminary Design Review (PDR) and Critical Design Review (CDR). A Physical Configuration Audit (PCA) is the formal examination of the "as-built" configuration of a configuration item against its technical documentation to establish or verify the configuration item's product baseline. The physical configuration audit done at the point of delivery establishes that all the products in the delivery baseline are actually in the delivery baseline and that they are the correct versions, as documented in the Version Description Document.

Software Assurance personnel should sign off on the delivery documentation to indicate that the delivery is complete and meets its requirements. More detail on the FCA and PCA are found in the software guidance in this requirement (SWE-084).

Examples of checklists for a functional configuration audit and a physical configuration audit are below: To get a downloadable copy of the checklists, click on the small document box.

Functional Configuration Audit Checklist  ^PAT-001

Click on the image to preview the file. From the preview, click on Download to obtain a usable copy. 

Physical Configuration Audit Checklist  ^PAT-002

Click on the image to preview the file. From the preview, click on Download to obtain a usable copy. 

7.5 Additional Guidance

Additional guidance related to this requirement may be found in the following materials in this Handbook: