Many resources exist to help a Mission Directorate develop the software measurement system it needs (see SWE-095). The NASA Software Measurement Guidebook is aimed at helping organizations begin or improve a measurement program. The Software Engineering Institute at Carnegie Mellon University has detailed specific practices for measurement and analysis within its CMMI-Dev, Version 1.3 model. The Software Technology Support Center (STSC), at Hill Air Force Base, has its "Software Metrics Capability Evaluation Guide" . Other resources are suggested in section 5 (Resources).
A Mission Directorate organization will establish a software measurement program for many reasons. Those range from having good management information for guiding software development to carrying out research toward the development of some innovative advanced technique. However, NASA has shown that the three key reasons for software measurement are to
- Understand and model software engineering processes and products.
- Aid in the management of projects.
- Guide improvements in software engineering processes.
The Mission Directorate plan will settle on and document the goals and objectives specifically chosen for its programs and projects. The collection of data measures are designed and dedicated to support the formation and analysis of metrics that describe the state and quality of the activities employed to execute the Mission Directorate's projects and programs. To emphasize this point, a quote from the CMMI-Dev document is cited here: "The measurement activities should support information needs at multiple levels including the business, organizational unit, and project to minimize re-work as the organization matures." The Mission Directorate can define its plan to be applicable to all activities of software development, while at the same time being specific to the specific nature of each project. The Mission Directorate will typically derive its measurement objectives from management, technical, project, software product or software process improvement needs. These objectives can be expected to evolve as the goals and objectives of the Mission Directorate change.
The following information is a brief synopsis of the planning, collection, storage and analysis activities to be performed by the Mission Directorates.
The guidance information presented in SWE-092, SWE-093, SWE-094, and SWE-095 for Center and project software measurement activities is meant to reflect the chosen Mission Directorate goals and objectives.
The general approach and primary steps for the Mission Directorate and the subsequent project level software measurement are very similar, with only the major objectives and goals expected to be unique. The program and mission support offices at the Centers contribute to many Mission Directorate needs for leading indicators with assessments of their internal capabilities against the specified program and project goals and desired outcomes.
Proper execution of data collection necessitates an agreed-to plan for the software measurement program. The plan is based on the evaluation and interpretation of specified software measures that have been captured, validated and analyzed according to the approved procedures in the software measurement plan. The activities begin with clear information that includes:
- A description of all data to be provided.
- A precise definition of terms.
- A description of responsibilities for data provision and analyses.
- A description of time frames and responsibilities for reception of the data and analyses being provided.
The data collection efforts by the Centers, projects, and software development teams can produce more accurate results if the time to collect the data is minimized. If the data collectors and analysis providers see this as a non-value added task, data collection will become sporadic and analysis quality will suffer.
Activities within the data storage procedure include:
- A description of the checking, validation and verification (V&V) of the quality of the data sets collected. This includes checks for proper formats, logicalness, missing entries, repetitive entrees, typical value ranges (expect these to be oriented towards validation at the set level since individual data checking and validation will have been already performed at the project level).
- A description of what data sets, or intermediate analyses will be made and kept, or made and discarded (assuming the analyses can be reconstructed if needed). This includes a listing of requested analyses by Mission Directorate stakeholders, lists of continuing metrics, and descriptions of changes to the analyses to reflect advances in the software development life cycle.
- The identification of a proper storage system, and site and management steps to access, use, and control the data and its appropriate data base management system (DBMS). The use of a DBMS allows multiple projects and organizations to access the data in a format that supports their specific or organizational objectives.
Metrics (or indicators) are computed from measures using the Mission Directorate's analysis procedures. They are quantifiable indices used to compare software products, processes, or projects or to predict their outcomes. They show trends of increasing or decreasing values, relative only to the previous value of the same metric. They also show containment or breeches of pre-established limits, such as allowable latent defects.
Note: Management metrics are measurements that help evaluate how well software development activities are performing across multiple Centers, development organizations, programs or projects. Trends in management metrics support forecasts of future progress, early trouble detection, and realism in plan adjustments of all candidate approaches that are quantified and analyzed.
Additional guidance related to software measurement determination, collection, analysis, and reporting may be found in the following related requirements in this handbook that are written from the project development point of view.