The Headquarters Office of the Chief Engineer (OCE) controls and maintains an appraisal process for use in periodic Center and project OCE compliance surveys. The OCE compliance survey achieves several objectives. They are:
- Review Center and specified NASA Headquarters organizations’ processes and infrastructure for compliance with OCE requirements, policy, procedures, processes, statutes, and regulations.
- Review specific program/project “files” for compliance with requirements, policy, procedures, processes, statutes, and regulations.
- Identify systemic problems or deficiencies.
- Recognize areas of excellence/best practices.
- Receive Center feedback regarding modifications in Agency policy and requirements.
Currently, the OCE compliance surveys focus on the following core elements:
- Common framework for unified program and project life cycle.
- Program and project review structure.
- Technical authority implementation.
- Dissenting opinions and deviation/waiver process.
- Software engineering management.
- Systems engineering.
- Lessons learned.
- Technical standards.
In addition to NPR 7150.2, the Headquarters’ OCE compliance survey may also include a review and appraisal of the products resulting from the use of the following documents, to the extent they involve software engineering:
- NPD 7120.4D, NASA Engineering and Program/Project Management Policy.
- NASA-STD-8739.8, Software Assurance Standard.
- NASA-STD-8719.13, Software Safety Standard.
The two NASA standards in this list are traditionally covered in detail by the Office of Safety and Mission Assurance (OSMA) audits conducted by the NASA Safety Center.
“The baseline set of questions are reviewed and may be revised as needed to support the survey at each specific organization. Input for updates to the questions is obtained from survey team members including the software engineering sub-team lead, the systems engineering sub-team lead, and the
Office of the Chief Information Officer (OCIO) representative for records management” (OCE Requirements Compliance Survey Process, 2010).
The process of determining the scope for a survey addresses the following items at a minimum:
- Requirements implementation and compliance.
- Results from audits, reviews, and assessments conducted by other organizations.
- Trends identified across the Agency or within a single organization.
Preparations for the survey typically include reviews of the flow down of NASA OCE requirements to Center/Project procedural documents, reviews of organization and program/project specific documentation and reviews of other surveys, audits and assessments. What follows in this guidance is a brief summary of the software engineering survey team's appraisal process. The main thrust of the software sub-team's appraisal is built into a set of questions from the OCE. This baseline set of questions serves as guidance to the Center or project to communicate what the OCE wants to review. The survey leader will communicate these questions to the Center's survey manager 3 to 4 weeks before the event, who in turn conveys them to the software point of contact. (SW POC). This is usually the Center's NASA Software Working Group (SWG) primary representative.
As in many reviews, surveys, and appraisal activities, compliance is often measured against objective evidence. Expected content and types of this objective quality evidence are usually defined in the preparation for the software survey. This defined material provides the basis for confirmation of compliance with requirements and identification of strengths and weaknesses. The Carnegie Mellon University Software Engineering Institute provides the Capability Maturity Model Integration (CMMI®) appraisal method for process improvement. The method provides an excellent discussion of the philosophy and types of objective evidence. The degree to which Centers follow the CMMI® method can be assessed in a Standard CMMI Appraisal Method for Process Improvement (SCAMPISM) activity. While the information presented in the SCAMPISM discussion is centered on evaluating CMMI® process implementation, the discussions and explanations in the CMMI® text provide good background information for use in the OCE appraisals to people for who are relatively inexperienced in appraisals and surveys.
Findings resulting from the survey are generally classified as strengths, weaknesses, observations, opportunities, and non-compliances. However, the survey team has a clear and overriding obligation to identify all items of non-compliance and items that adversely affect safety or quality. These items will be included in the final report. Significant issues are brought to the immediate attention of the surveyed organization's management via the survey manager
Additional guidance related to OCE Appraisal activities may be found in the following related requirements and sections in this Handbook: