bannera

Book A.
Introduction

Book B.
7150 Requirements Guidance

Book C.
Topics

Tools,
References, & Terms

SPAN
(NASA Only)


SWE-090 - Measurement Objectives

1. Requirements

4.4.1 The project shall establish and document specific measurement objectives for their project.

1.1 Notes

NPR 7150.2, NASA Software Engineering Requirements, does not include any notes for this requirement.

1.2 Applicability Across Classes

This requirement has modified applicability for the following classes:

  • Classes D-E and Safety Critical are labeled "X +SO." This means that this requirement applies to the safety-critical aspects of the software.
  • Class F is labeled with "X (not OTS)." This means that this requirement does not apply to off-the-shelf (OTS) software for these classes.
  • Class G is labeled with "P (Center)." This means that a Center defined process which meets a non-empty subset of this full requirement can be used to meet the intent of this requirement.

Class

  A_SC 

A_NSC

  B_SC 

B_NSC

  C_SC 

C_NSC

  D_SC 

D_NSC

  E_SC 

E_NSC

     F      

     G      

     H      

Applicable?

   

   

   

   

   

   

    X

   

    X

   

    X

    X

   

Key:    A_SC = Class A Software, Safety Critical | A_NSC = Class A Software, Not Safety Critical | ... | - Applicable | - Not Applicable
X - Applicable with details, read above for more | P(C) - P(Center), follow center requirements or procedures

2. Rationale

Measurement objectives document the reasons for doing software measurement and the accompanying analysis. They also provide a view into the types of actions or decisions that may be made based on results of analysis. Before implementing a measurement program or choosing measures for a project, it is important to know how the measures are going to be used. Establishing measurement objectives in the early planning stages helps ensure organizational and project information needs are being met and that the measures that are selected for collection will be useful.

There is some cost to collecting and analyzing software measures, so it is desirable to keep the measurement set to the minimum that will satisfy information needs. When measures are tied to objectives, a minimum useful set can be easily selected. Defining measurement objectives helps to select and  prioritize the candidate measures to be collected. If a measurement isn't tied to an objective, it probably doesn't need to be collected.

3. Guidance

Measurement objectives document the reasons why software measurement and analysis are performed. Measurement objectives consider both the perspectives of the organization as well as  the project. The sources for measurement objectives may be management, technical, project, product or process implementation needs, such as:  

  • Deliver software by the scheduled date.
  • Improve cost estimation.
  • Complete projects within budget.
  • Devote adequate resources to each process area.

Corresponding measurement objectives for these needs might be:

  • Measure project progress to ensure it is adequate to achieve completion by the scheduled date.
  • Measure the actual project planning parameters against the estimated planning parameters to identify deviations.
  • Track the project cost and effort to ensure project completion within budget.
  • Measure resources devoted to each process area to ensure they are sufficient.

According to NPR 7150.2, NASA's software measurement programs are designed to meet the following high-level goals:

  • To improve future planning and cost estimation.
  • To provide realistic data for progress tracking.
  • To provide indicators of software quality.
  • To provide baseline information for future process improvement.

Centers may further refine these objectives or add some Center-specific objectives. The measurement objectives a project defines are to be based on their Center's objectives and NASA's objectives. Usually project objectives are focused on the information needs the project has to provide information for managing and controlling the project. When establishing measurement objectives, ask what questions will be answered with the data, why you are measuring something and what types of decisions will be made with the data. A project may also have additional objectives to provide information to their Center or to NASA. This information may be used for process improvement or for developing the organizational baselines and trends. For example, Goddard Space Flight Center (GSFC) has defined two high-level objectives for its software projects as follows:

  • To assure that the effort is on track for delivery of the required functionality on time and within budget.
  • To improve software development practices both in response to immediate customer issues and in support of the organizational process improvement goals.

As you can see, one of these objectives focuses on the project's ability to manage and control the project with quantitative data and the other objective focuses on organizational process improvement.

In addition, GSFC's Measurement Planning Table Tool found in the Tools listing in the Resources section of this SWE, lists the following more specific measurement objectives for their projects:

  • Ensure schedule progress is within acceptable bounds.
  • Ensure project effort and costs remain within acceptable bounds.
  • Ensure project issues are identified and resolved in a timely manner.
  • Deliver the required functionality.
  • Ensure the performance measures are within margins.
  • Ensure that the system delivered to operations has no critical or moderate severity errors.
  • Minimize the amount of rework due to defects occurring during development.
  • Ensure requirement are complete and stable enough to continue work without undue risk.
  • Support future process improvement.

Projects typically check with their Center to ensure they have chosen objectives that support their Center-level objectives. 

4. Small Projects

Since small projects are typically constrained by budget and staff, they choose the objectives most important to them to help keep the cost and effort within budget. Some Centers have tailored measurement requirements for small projects. Be sure to check your Center's requirements when choosing objectives and associated measures.

Often small projects have difficulty affording  tools that would enable automatic measurement collection. There are several solutions for this issue. Some Centers, such as GSFC, have developed simple tools (often Excel-based) that will produce the measurements automatically. GSFC examples are the staffing tool, the requirements metrics tool, the action item tool, the risk tool, and the problem reporting tool. More information about these tools can be found in the Tools section under the Resources tab in this SWE.

Other solutions for small projects involve organizational support. Some organizations support a measurement person on staff to assist the small projects with measurement collection and some Centers use tools that can be shared by the small projects.

5. Resources

5.1 Tools

Tools relative to this SWE may be found in the table below. You may wish to reference the Tools Table in this handbook for an evolving list of these and other tools in use at NASA. Note that this table should not be considered all-inclusive, nor is it an endorsement of any particular tool. Check with your Center to see what tools are available to facilitate compliance with this requirement.

Tool nameTypeOwner/SourceLinkDescriptionUser

SpotCheck

Custom Software

KSC POC: Brian Bateman

...

Database/excel tool used to plan and track software activities. Reports allow % complete reporting, The Spot Check system provides software and spreadsheets to facilitate the collection of reporting status as well as the generation of unique and periodic status reports. The SpotCheck system attempts to address the following goals: *Reduce the effort associated with reporting status and issues. *Provide focus to the development team on priority work. *Capture all the development work being done. *Account for Time Away and Other Duties.

KSC

Problem Report Tool

Downloadable

GSFC

http://software.gsfc.nasa.gov/toolsDetail.cfm?selTool=2.5.2.3 ...

v1.0, Excel-based problem report management and metrics tool, GSFC

GSFC

Staffing Tool

Downloadable (for Excel 2007 only) SPAN - Accessible to NASA users via SPAN tab in this Handbook. By Request - Non-NASA users, contact User for a copy of this tool.

GSFC

...

This tool is used to plan staffing resources and track actual and projected effort against the plan. This tool is also used to plan procurement costs and track actual expenditures against the plan. This is downloadable in Excel 2007 only. Search in SPAN for "GSFC_TL_20150126_Staffing_Tool".

GSFC

Risk Management Tool

SPAN - Accessible to NASA users via SPAN tab in this Handbook. By Request - Non-NASA users, contact User for a copy of this tool.

GSFC

...

Provides a means for projects to specify and monitor risks. It supports up to 30 risks. Information tracked includes the statement of the risk, originator, date identified, probability, impact, timeframe, assignee, visibility, source, and mitigation steps. This Tool generates detail and summary reports. Search in SPAN for "GSFC_TL_20120905_Risk_Mgmt_Tool"

GSFC

RequirementsLink

COTS

ENSER/Parametric Technology Corporation (PTC)

http://www.ptc.com/appserver/wcms/relnotes/note.jsp?icgdbkey=826imdbkey=119829 ...

Windchill RequirementsLink. Requirements capture and tracking tool. Windchill RequirementsLink - an integral option for Windchill PDMLink - lets you manage product requirements, including change control and associating requirements with specific product structures and design content. With bi-directional traceability between customer needs, market requirements and the underlying technical requirements, you can ensure that customer and market requirements are satisfied by designs, and properly verified during development.

SSC

Requirements Metrics Tool

SPAN - Accessible to NASA users via SPAN tab in this Handbook. By Request - Non-NASA users, contact User for a copy of this tool.

GSFC

...

The requirements metrics spreadsheet is used to track both functionality (via the number of requirements representing the scope of the system) and requirements volatility (by tracking changes to requirements). It has three tabs for input data and calculations, and four tabs for graphs of said data. The inputs are project information that helps set up the spreadsheets, data allocating requirements to build and CSCI, and a timeline of requirements changes that tracks the evolving number of requirements and requirements changes by CSCI. Search in SPAN for "GSFC_TL_20070501_Req_Metrics_Tool"

GSFC

Measurement Planning Table Tool

SPAN - Accessible to NASA users via SPAN tab in this Handbook. By Request - Non-NASA users, contact User for a copy of this tool.

GSFC

...

Provides a template for both development and acquisition projects for specifying the measures that should be collected over the project life cycle. For each measurement area (e.g., Software Quality), the template provides suggestions for the measurement objectives, the measurements that should be collected, the collection frequency, and the analysis that should be performed. Search in SPAN for "GSFC_TL_20160909_Measurement_Planning_Table_Tool"

GSFC

DOORS®

COTS

IBM® Rational®

http://www-01.ibm.com/software/awdtools/doors/ ...

IBM® Rational® DOORS® family is a group of requirements management tools that allow you to capture, trace, analyze and manage changes across the development lifecycle.

ARC, DFRC, GRC, GSFC, IV&V, JPL, JSC, JSC, LaRC, MSFC,

CodeSonar®

COTS

Grammatech

http://www.grammatech.com/products/codesonar/overview.html ...

By analyzing both source code and binaries, CodeSonar enables teams to analyze complete applications, enabling you to take control of your software supply chain and eliminate the most costly and hard-to-find defects early in the SDLC.

IV&V, ARC (NanoSat), JPL

Action Item Tracking Tool

SPAN - Accessible to NASA users via SPAN tab in this Handbook. By Request - Non-NASA users, contact User for a copy of this tool.

GSFC

...

Excel spreadsheet that tracks action items and produces a summary report. Attributes tracked for each action item include ID, Action Item, Assigned To, Priority, Date Opened, Date Due, Date Closed, Days Opened, and Notes. Available in SPAN on page: GSFC_TL_20080905_Action_Item_Tracking

GSFC

6. Lessons Learned

The NASA Lesson Learned database contains lessons learned addressing topics that should be kept in mind when planning a software measurement program. The lessons learned below are applicable when choosing objectives and related measures:  

  • Know How Your Software Measurement Data Will Be Used. Lesson Number 1772: When software measurement data used to support cost estimates is provided to NASA by a project without an understanding of how NASA will apply the data, discrepancies may produce erroneous cost estimates that disrupt the process of project assessment and approval. Major flight projects should verify how NASA plans to interpret such data and use it in their parametric cost estimating model, and consider duplicating the NASA process using the same or a similar model prior to submission. 567
  • Space Shuttle Processing and Operations Workforce. Lesson Number 1042: Operations and processing in accordance with the Shuttle Processing Contract (SPC) have been satisfactory. Nevertheless, lingering concerns include: the danger of not keeping foremost the overarching goal of safety before schedule before cost; the tendency in a success-oriented environment to overlook the need for continued fostering of frank and open discussion; the press of budget inhibiting the maintenance of a well-trained NASA presence on the work floor; and the difficulty of a continued cooperative search for the most meaningful measures of operations and processing effectiveness. 583

    A recommendation from Lesson Number 1042 states "NASA and SPC should continue to search for, develop, test, and establish the most meaningful measures of operations and processing effectiveness possible."