1. Requirements

4.4.7 Each NASA Mission Directorate shall identify and document the specific measurement objectives, the chosen specific measures, the collection procedures, and storage and analysis procedures.

1.1 Notes

NPR 7150.2, NASA Software Engineering Requirements, does not include any notes for this requirement.

1.2 Applicability Across Classes

2. Rationale

Each Mission Directorate develops its own software measurement system (see SWE-095) and tailors it to its needs and objectives. The system is based on an understanding of the development environment used for executing its programs and projects. Increased understanding of software progress leads to better oversight of projects. Mission Directorate specified analysis procedures allow the synthesis of results across Centers and across performing organizations, and within programs composed of one or more projects.

While much of the software measurement data and the checking and validation activities may be captured and performed at the project level, the information that is transmitted to the Mission Directorate is more readily used if it is provided in specified formats that allow the Mission Directorate to satisfy its review and oversight needs. Subsequent evaluation and interpretation of these software measurements enable the Mission Directorate to assess its current software status and engineering capabilities of providers for future work.

3. Guidance

Many resources exist to help a Mission Directorate develop the software measurement system it needs (see SWE-095). The NASA Software Measurement Guidebook ³²⁹ is aimed at helping organizations begin or improve a measurement program. The Software Engineering Institute at Carnegie Mellon University has detailed specific practices for measurement and analysis within its CMMI-Dev, Version 1.3 ¹⁵⁷ model. The Software Technology Support Center (STSC), at Hill Air Force Base, has its "Software Metrics Capability Evaluation Guide" ³³⁶. Other resources are suggested in section 5 (Resources).

A Mission Directorate organization will establish a software measurement program for many reasons. Those range from having good management information for guiding software development to carrying out research toward the development of some innovative advanced technique. However, NASA ³²⁹ has shown that the three key reasons for software measurement are to

1. Understand and model software engineering processes and products.
2. Aid in the management of projects.
3. Guide improvements in software engineering processes.

The Mission Directorate plan will settle on and document the goals and objectives specifically chosen for its programs and projects. The collection of data measures are designed and dedicated to support the formation and analysis of metrics that describe the state and quality of the activities employed to execute the Mission Directorate's projects and programs. To emphasize this point, a quote from the CMMI-Dev document is cited here: "The measurement activities should support information needs at multiple levels including the business, organizational unit, and project to minimize re-work as the organization matures." ¹⁵⁷ The Mission Directorate can define its plan to be applicable to all activities of software development, while at the same time being specific to the specific nature of each project. The Mission Directorate will typically derive its measurement objectives from management, technical, project, software product or software process improvement needs. These objectives can be expected to evolve as the goals and objectives of the Mission Directorate change.

The following information is a brief synopsis of the planning, collection, storage and analysis activities to be performed by the Mission Directorates.

The general approach and primary steps for the Mission Directorate and the subsequent project level software measurement are very similar, with only the major objectives and goals expected to be unique. The program and mission support offices at the Centers contribute to many Mission Directorate needs for leading indicators with assessments of their internal capabilities against the specified program and project goals and desired outcomes.

Proper execution of data collection necessitates an agreed-to plan for the software measurement program. The plan is based on the evaluation and interpretation of specified software measures that have been captured, validated and analyzed according to the approved procedures in the software measurement plan. The activities begin with clear information that includes:

• A description of all data to be provided.
• A precise definition of terms.
• A description of responsibilities for data provision and analyses.
• A description of time frames and responsibilities for reception of the data and analyses being provided.

The data collection efforts by the Centers, projects, and software development teams can produce more accurate results if the time to collect the data is minimized. If the data collectors and analysis providers see this as a non-value added task, data collection will become sporadic and analysis quality will suffer.

Activities within the data storage procedure include:

• A description of the checking, validation and verification (V&V) of the quality of the data sets collected. This includes checks for proper formats, logicalness, missing entries, repetitive entrees, typical value ranges (expect these to be oriented towards validation at the set level since individual data checking and validation will have been already performed at the project level).

• A description of what data sets, or intermediate analyses will be made and kept, or made and discarded (assuming the analyses can be reconstructed if needed). This includes a listing of requested analyses by Mission Directorate stakeholders, lists of continuing metrics, and descriptions of changes to the analyses to reflect advances in the software development life cycle.

• The identification of a proper storage system, and site and management steps to access, use, and control the data and its appropriate data base management system (DBMS). The use of a DBMS allows multiple projects and organizations to access the data in a format that supports their specific or organizational objectives.

Metrics (or indicators) are computed from measures using the Mission Directorate's analysis procedures. They are quantifiable indices used to compare software products, processes, or projects or to predict their outcomes. They show trends of increasing or decreasing values, relative only to the previous value of the same metric. They also show containment or breeches of pre-established limits, such as allowable latent defects.

Additional guidance related to software measurement determination, collection, analysis, and reporting may be found in the following related requirements in this handbook that are written from the project development point of view.

4. Small Projects

This SWE requirement indirectly applies to projects, in that a project's software data feeds into the Mission Directorate's measurement system. The amount of data needed by Mission Directorates to monitor software risks on small projects is likely to be considerably less than for larger projects.

5. Resources

5.1 Tools

Tools to aid in compliance with this SWE, if any, may be found in the Tools Library in the NASA Engineering Network (NEN).

NASA users find this in the Tools Library in the Software Processes Across NASA (SPAN) site of the Software Engineering Community in NEN.

The list is informational only and does not represent an “approved tool list”, nor does it represent an endorsement of any particular tool. The purpose is to provide examples of tools being used across the Agency and to help projects and centers decide what tools to consider.

6. Lessons Learned

A documented lesson from the NASA Lessons Learned database notes the following:

Know How Your Software Measurement data will Be used, Lesson No. 1772: "Prior to Preliminary Mission & Systems Review (PMSR), the Mars Science Laboratory (MSL) flight project submitted a Cost Analysis Data Requirement (CADRe) document to the IPAO (Independent Program Assessment Office) that included an estimate of source lines of code (SLOC) and other descriptive measurement data related to the proposed flight software. The IPAO input this data to their parametric cost estimating model. The project had provided qualitative parameters that were subject to misinterpretation, and provided physical SLOC counts. These SLOC values were erroneously interpreted as logical SLOC counts, causing the model to produce a cost estimate approximately 50 percent higher than the project's estimate. It proved extremely difficult and time-consuming for the parties to reconcile the simple inconsistency and reach agreement on the correct estimate."

The Recommendation states that "Prior to submitting software cost estimate support data (such as estimates of total SLOC and software reuse) to NASA for major flight projects (over $500 million), verify how the NASA recipient plans to interpret the data and use it in their parametric cost estimating model. To further preclude misinterpretation of the data, the software project may wish to duplicate the NASA process using the same or a similar parametric model, and compare the results with NASA's." ⁵⁶⁷