3. GuidanceThe key parts of the requirement that need to be addressed as a collaborative effort among software project managers and Center project and cost offices are: - Establish a software cost estimation process. Preferably a project will use an industry standard process or tool like Constructive Cost Model (COCOMO™) or Software Evaluation and Estimation of Resources - Software Estimating Model (SEER® SEM) tailored for the organization performance and parameters.
- Document the process, parameters and results of the software estimations. Maintain records of the software estimates and data associated with uncertainly and assumptions for the given estimate.
- Document and maintain all associated software cost parameters. The ability to reproduce the software estimate and understand the risk associated with the estimate is generally contained in the software cost model parameters.
- The cost estimate covers the entire lifecycle given the starting point of the estimate.
- Cost estimates are developed and maintained over the software lifecycle including any time the schedule or resource allocations change or requirements change.
- Different software cost estimates are developed for the different classes of software on a project and for different organizations developing software on a project. Software cost parameters are maintained for each software cost estimate.
As software’s importance in missions has grown, the focus on its overall performance both technically and programmatically has also increased. As a result, software has been blamed for launch slips, mission failures, and contributing to major project cost growth. Estimation errors resulting in software cost growth rates of 50 percent to over a 100 percent are typically identified as a contributing factor in post development lessons learned. Barry Boehm's  seminal chart shown below captures one of the main reasons why estimation is difficult. The figure illustrates that, estimation uncertainty is so large in the early stages of the life cycle, that estimates can be off by a factor of four. 
Fortunately, 30 years of cost estimation and modeling research and practice have identified methods for addressing these known issues. These are reflected in the process described in this requirement. Software cost estimates are a key activity performed as part of software life-cycle planning (NPR 7150.2, Section 3.1) with regular updates made throughout the entire lifecycle. This activity is performed as a part of software tasks of all sizes and all classes with the primary difference being the level of detail and amount of review of the estimates. Two software cost estimates are required for software class A and B projects with an estimated project cost of 2 million dollars or more. Two estimates are also recommended when project scope is not well understood. It is highly recommended that model-based and analogy estimates be completed as verification or backup to the primary estimate, which is typically a bottom-up estimate. Multiple estimates are especially important due to the extensive estimation uncertainty that exists in the early part of the lifecycle. Most models provide a method for expressing the estimate uncertainty that the project team can use as part of the basis of estimate. The key elements of the process are the development of the following: - Two estimates, one of which is a model based estimate.
- A software size estimate and other key planning parameters.
- A risk list with mitigations (SWE-151).
- A documented basis of estimate (BOE) that can be revised as plans are updated.
Scope the Job The project team should determine the scope of the software task, including what the task is and is not, and what it covers. When the project manager collaborates with Center project and cost offices to generate cost estimates, the results of scoping the work is a key input from the project. Scoping is not requirements writing – requirements are generally developed after initial planning and estimating is completed. |
The goal is to identify clearly and in sufficient detail the entire set of software products and activities for producing a good cost estimate. This includes understanding the intended concept of operations and operating environment to fully characterize the complexity of the work. The project team also identifies values for key planning parameters, including the systems schedule, risk posture, new technology assumptions, expected inheritance, and complexity. Typically, the scope of a software development effort includes all activities associated with software management, software engineering, programming, and software test engineering over the software life cycle. Included in this effort are software requirements analysis, software design, software implementation, and software integration and testing. Bottom-Up Estimate When developing the bottom-up cost estimate, whether at the project level or in conjunction with the appropriate Center offices, the first step is to develop the work breakdown structure (WBS). A critical aspect of developing the WBS is ensuring that it is easily mapped to other key task breakdowns, including the functional or object decomposition and schedule elements. It is important to be able to quickly modify the cost, schedule and functional design as changes occur to keep them consistent. A recommended sequence of steps for the bottom-up estimate is the following. - Define a WBS.
- Estimate size and/or effort for each element.
- If available, map analogous size, effort and/or costs into current WBS.
- Adjust baseline for your mission and document how and why it was adjusted as part of the bottom-up estimate.
- Lay the estimate out over time.
- Use approved tool to get costs.
- Use reference data to assist in effort estimate or to check distribution of effort.
Estimate Software Size Software size is a required input for the use of a cost model and can also be used as part of a bottom-up estimate. Estimating software size is one the most complicated estimation activities. Software size can be measured in various ways but the most common measure is to count lines of code (LOC). The most used LOC metrics are physical lines (carriage returns excluding comments and blanks) and logical lines, which counts logical delimiters such as terminal semi-colons in C along with preprocessor directives and terminal close-braces. The accuracy of size estimate is greatly improved when anchored by actual code counts from similar software products or similar software components, which require a code counter such as Source Lines Counter (SLiC), found on the NASA Software PAL (See the  ). It is also required to distinguish between new, reused and modified-reused code as the cost of developing a code is very different for each type of code. A common estimation mistake is the over-estimation of cost savings when reused and modified-reused software is included in a system. Care should be taken not to overestimate the amount of reused code or underestimate the effort to modify that code for use in the system. Incorporating reused code requires the developer to do more than just testing and may not necessarily result in significant cost savings. An assessment of the code may be needed to determine how much code can actually be used in the new system. Code modification is necessary and that effort is affected by several factors including how effectively the code can be understood (i.e., how well it is commented, how well it is structured, etc.), the amount of documentation that exists for the reused code, whether the code was designed for reuse, etc. One recommendation is that if the reused code will require greater than 50 percent modifications, then it should be treated as new code for cost estimation purposes. |
For more information and a detailed description of how to handle the different types of code, consult Software Engineering Economics, by Boehm , Software Cost Estimation with COCOMO II, by Boehm, et. al.  , and the Handbook for Software Cost Estimation by Lum, et. al., (accessible to NASA users in Software Processes Across NASA (SPAN) from the SPAN tab in this Handbook). Other methods that can be used to estimate the software size include function point and use case counting. Model-Based Estimates Model-based estimates are estimates made using parametric cost models. Parametric cost models have been in use at NASA for over 20 years. The two most used parametric software cost models are COCOMOTM II and SEER®-SEM. The Software COCOMO™ Analysis Tool (SCAT) is a probabilistic version of COCOMOTM II and can be found on the NASA Software PAL. (Refer to the for additional information on these tools.) Cost models can be used as a primary estimate early in the life cycle, as a secondary backup estimate for validation, and to help you reason about the cost and schedule implications of software decisions you may need to make. Cost models require an estimate of the size of the system and a set of inputs describing the system and development environment; these items are referred to as cost drivers or effort multipliers. Effort multipliers characterize the product, platform, personnel, and project attributes of the software project under development. |
As an example, COCOMOTM II has 17 effort multipliers. Each of the COCOMOTM parameters is associated with up to six ratings – “very low,” “low,” “nominal,” “high,” “very high,” and “extra high”, which correspond to a real number based upon the factor and the degree to which the factor can influence productivity. A rating equal to 1 does not increase nor decrease the schedule and effort (this rating is called “nominal”). A rating less than 1 denotes a factor that can decrease the schedule and effort. A rating greater than 1 denotes a factor that increases the schedule or effort. For a detailed description of the COCOMOTM model and the user guides for the commercial cost models, consult Software Engineering Economics, by Boehm , Software Cost Estimation with COCOMO II, by Boehm, et. al. , and the Handbook for Software Cost Estimationby Lum, et. al. accessible to NASA users in SPAN. Validation and Reconciliation The purpose of this step is to review the validated estimates with respect to the project-imposed budget and schedule, and to resolve the differences. If an inconsistency arises, there is a tendency to incorrectly address the issue as only a problem of incorrect estimation. However, in most cases, the real solution is to reduce scope or functionality, and then to reduce scope again, until the task fits the budget. This step is another area where the project can provide valuable input to Center costing offices since the project team has the best understanding of the work to be performed. Do not reduce costs by eliminating reserves and making optimistic and unrealistic assumptions, especially with respect to the amount of code inheritance. |
Review, Approve, and Maintain the Estimates The purpose of this step is to review the software estimates and to obtain stakeholder approval. The project team should conduct a peer review with the following objectives: - Confirm the scope and Work Breakdown Structure (WBS) are complete and accurate.
- Verify the methods used for deriving the size, effort, schedule, and cost. • Ensure the assumptions and input data used to develop the estimates are captured in the cost models and are correct, • Ensure that the estimates are reasonable and accurate, given the input data.
- Formally confirm and record the approved software estimates and underlying assumptions for the project.
The stakeholders, which include the software manager, software estimators, line management, and project management, approve the software estimates after the review is complete and problems have been resolved. Remember that costs cannot be reduced without reducing functionality. The approved software estimates and supporting information such as the BOE should be configuration managed in a location accessible to stakeholders. The estimates should be updated in response to changes in schedule, budget, technical scope, development methodology, personnel or other factors that could alter the accuracy of the approved estimates. If the project has a cost repository, that repository should be updated with data related to this cost estimate, including software size, effort, and schedule data. NASA-specific cost estimation process information is available in Software Processes Across NASA (SPAN), accessible to NASA users from the SPAN tab in this Handbook. Additional guidance related to software cost estimation may be found in the following related requirements in this Handbook: |