- 1. The Requirement
- 2. Rationale
- 3. Guidance
- 4. Small Projects
- 5. Resources
- 6. Lessons Learned
- 7. Software Assurance
5.1.9 The project manager shall participate in any joint NASA/developer audits.
NPR 7150.2, NASA Software Engineering Requirements, does not include any notes for this requirement.
Click here to view the history of this requirement: SWE-045 History
1.3 Applicability Across Classes
Key: - Applicable | - Not Applicable
A & B = Always Safety Critical; C & D = Sometimes Safety Critical; E - F = Never Safety Critical.
Per IEEE Std 1028-2008, Software Reviews and Audits, "The purpose of a software audit is to provide an independent evaluation of conformance of software products and processes to applicable regulations, standards, guidelines, plans, specifications, and procedures." 219 Audits are part of the supplier/provider monitoring activities performed by the acquirer 224, but may also be external audits, internal audits, or some other type of audit.
Audits are conducted by audit teams and require the participation and cooperation of the personnel involved with the software being audited, both acquirer and provider personnel, including contractors, as appropriate for the particular audit being performed.
This requirement is not intended to force joint audits, but when audits occur, the project needs to be made aware of and participate at some level in those audits, whether they are internal audits, contractor audits, external audits by an independent organization, or any other type of internal or external audit. Project participation can benefit the audit by providing domain knowledge, planning assistance, and technical expertise to the audit team.
This requirement was written to require projects to participate in audits that include any or all of the software portion of a project. The project's participation can take many forms, including, but not limited to, simply keeping abreast of the audit's progress as well as participating as an observer in the actual audit.
It is the responsibility of the project to make available appropriately prepared and qualified project personnel to participate or support audits as needed to fulfill the project's chosen level of involvement, including software assurance personnel described in the project's software assurance plan (see NASA-STD-8739.8 278, Software Assurance and Software Safety Standard, for software assurance involvement in audits).
Consult Center Process Asset Libraries (PALs) for Center-specific guidance related to joint audits, particularly Project Monitoring and Control (PMC) documentation.
See topic 7.3 - Acquisition Guidance in this Handbook for additional guidance. Additionally, guidance related to joint audits may be found in the following related requirements in this Handbook:
4. Small Projects
For projects with limited personnel, consider limiting the audit participation in monitoring progress and reviewing the results as this would cause less interference and requires resources.
6. Lessons Learned
6.1 NASA Lessons Learned
The NASA Lesson Learned database contains the following lesson learned related to joint audits:
- Acquisition and Oversight of Contracted Software Development. Lesson Number 0921 528: "The loss of Mars Climate Orbiter (MCO) was attributed to, among other causes, the lack of a controlled and effective process for acquisition of contractor-developed, mission-critical software. NASA Centers should ... assure ... verification of the adequacy of the software design approach and overall contractor implementation throughout the software life cycle." Audits are one way to assess the adequacy of contractor implementation throughout the software life cycle."
6.2 Other Lessons Learned
No other Lessons Learned have currently been identified for this requirement.
7. Software Assurance
7.1 Tasking for Software Assurance
- Participate in or assess the results from any joint NASA/developer audits. Track any findings to closure.
7.2 Software Assurance Products
- Assessment of Joint NASA/developer Audit Results
Definition of objective evidence
- Provide confirmation that Task 1 has been done
Objective evidence is an unbiased, documented fact showing that an activity was confirmed or performed by the software assurance/safety person(s). The evidence for confirmation of the activity can take any number of different forms, depending on the activity in the task. Examples are:
- Observations, findings, issues, risks found by the SA/safety person and may be expressed in an audit or checklist record, email, memo or entry into a tracking system (e.g. Risk Log).
- Meeting minutes with attendance lists or SA meeting notes or assessments of the activities and recorded in the project repository.
- Status report, email or memo containing statements that confirmation has been performed with date (a checklist of confirmations could be used to record when each confirmation has been done!).
- Signatures on SA reviewed or witnessed products or activities, or
- Status report, email or memo containing Short summary of information gained by performing the activity. Some examples of using a “short summary” as objective evidence of a confirmation are:
- To confirm that: “IV&V Program Execution exists”, the summary might be: IV&V Plan is in draft state. It is expected to be complete by (some date).
- To confirm that: “Traceability between software requirements and hazards with SW contributions exists”, the summary might be x% of the hazards with software contributions are traced to the requirements.
- None at this time
Audits provide management with information about the project team, the project processes, and help identify best practices and areas of improvement.
Audits are useful to assess:
- Adequacy of project plans, processes, systems
- Compliance to those plans, processes, systems
- Effectiveness of those plans, processes, systems, and internal project controls on those processes
- Product fitness for use/compliance to specifications
- Areas for improvement
The results of audits allow project management to make adjustments and corrections to ensure high-quality products are being produced and delivered and that the team is functioning efficiently and effectively. Trending audit results over time allow management to identify systemic issues and areas of risk while monitoring the effect of process and product improvements.
Software assurance personnel should either perform or participate in any audits that NASA or the project does jointly with a developer organization. NASA software assurance personnel will also be doing insight/oversight on any project providers and confirming that they are compliant with NPR 7150.2 to the extent specified in their contract. Providers should be performing audits against their procedures, plans, and activities and NASA software assurance should be participating, or at the very least, reviewing the results and tracking the findings to closure. Any findings from audits that software assurance performs or participates in should be tracked to closure.
Track Findings to closure
After the audit report is delivered, the audit team continues to have closure responsibilities such as those listed below.
- Review corrective action (CA) plans or responses from the project in accordance with an agreed-upon timeframe with the project. Two weeks to 30 days is a reasonable timeframe depending on the project’s schedule and when they can reasonably implement the solutions, once approved.
- Review CA plans for expected content:
- Problem/Issue/Finding statement
- Root Cause investigation plan, including “where else does this need to be applied” and a due date
- Short term correction plan and a due date
- Long term CA plan (plan to avoid recurrence) and a due date
- Assess any provided rationale for why corrective action is not needed, e.g.:
- The project provided more evidence
- Project clarified an audit team misunderstanding
- Assess timeliness of CA plans, coverage of the associated Findings, and recurrence prevention plan
- Work any concerns with the project; this responsibility lies with the Lead Auditor and project manager
- Review CA plans for expected content:
- Once the project has implemented the CA plans, review the results to assess how well those plans addressed the relevant audit Findings. This includes a review of any revised documents, an assessment of revised process implementation, and perhaps a follow-up audit.
- When Findings are satisfactorily closed, the Lead Auditor notifies the project and audit team management in writing and captures the written notifications in the official audit records, including the rationale for closure of each Finding.
For information see the software assurance topic on software auditing.