bannerd


SWE-037 - Software Milestones

1. Requirements

3.1.7 The project manager shall define and document the milestones at which the software developer(s) progress will be reviewed and audited. 

1.1 Notes

NPR 7150.2, NASA Software Engineering Requirements, does not include any notes for this requirement.

1.2 History

SWE-037 - Last used in rev NPR 7150.2D

RevSWE Statement
A

2.5.6 The project shall define the milestones at which the software supplier(s) progress will be reviewed and audited as a part of the acquisition activities.

Difference between A and B

No change

B

3.12.6 The project manager  shall define the milestones at which the software supplier(s) progress will be reviewed and audited as a part of the acquisition activities.

Difference between B and C

Added the requirement to document the milestones; Changed "supplier(s)" to "developer(s)"; Descoped the reqts by removing "as a part of the acquisition activities"

C

3.1.7 The project manager shall define and document the milestones at which the software developer(s) progress will be reviewed and audited. 

Difference between C and DNo change
D

3.1.7 The project manager shall define and document the milestones at which the software developer(s) progress will be reviewed and audited. 



1.3 Applicability Across Classes

Classes F and G are labeled as "X (not OTS)" which means that the project is required to meet this requirement for all software that is not considered off-the-shelf.

Class

     A      

     B      

     C      

     D      

     E      

     F      

Applicable?

   

   

   

   

   

   

Key:    - Applicable | - Not Applicable


2. Rationale

For any software project, it is critical for management to review progress early and periodically to ensure the project remains on schedule, is progressing toward implementation of the requirements, and ultimately is addressing the customer's needs. It is also important for management to confirm periodically that the technical goals of the project are being achieved and that the technical direction of the project is appropriate (NPR 7123.1, NASA Systems Engineering Processes and Requirements). 041 Milestone reviews provide this type of management visibility into a project.

For software development that is acquired (supplied by a contractor), having regular progress reviews is even more important since these reviews are the keys to ensuring the contractor understood and will provide the product that NASA requested and that meets NASA's requirements for safety, quality, reliability, etc.

Milestone reviews can also serve to facilitate and ensure coordination between multiple development groups including development groups at multiple NASA Centers and contractors.

3. Guidance

The purpose of the milestone review meeting is to review the overall milestone progress and accomplishments at selected milestones/phases during the project life cycle. Milestone reviews are generally defined by the project manager. This meeting also facilitates the project team to have a get-together with higher management and other stakeholders and to discuss project issues; share lessons learned and suggest improvements for the next milestones.

3.1 Why Do Milestones Matter?

The use of project milestones is essentially a scheduling strategy. In smaller projects, you may only have two milestones: the project start and end dates. Milestones are also less common in Agile projects as the Agile Methodology focuses on short iterations and evolving schedules and plans. Within an Agile project, the only milestones may be at the end of each sprint. Milestones are more common in traditional project management methodologies that follow a more structured approach to planning and scheduling.

Milestones help ensure that critical project deadlines are met. Often these deadlines are dictated to you by the customer or project sponsor. For example, if you are developing software to support a space flight project the project manager would likely tell you when they want to see the first draft and when they expect to receive the final product. Both of these dates are milestones.

Milestones can also act as mile markers on long projects. For instance, let’s say you have a year-long project to complete a new product release. We can assume that the work effort is estimated to be fairly steady throughout the entire project. This means that, at the three-month mark, the project should be 25% complete, at the six-month mark it should be 50% complete, and at the nine-month mark, it should be 75% complete.

A milestone may be created at each of these points to assess actual progress compared with expected progress. If you arrive at the three-month mark to discover the project is only 15% complete, you now have an early warning sign that there may be a problem and the expected end date might be at risk.

Milestones are also useful for communicating project success. When you’re in the middle of a complex project, it can be difficult to determine who is accomplishing what. You can become so focused on the minutiae of each task that you lose perspective.

But with milestones, you can easily move away from the task level and see the overall progress more clearly. Plus, regularly communicating the milestones you’ve met is a great way to gain trust and keep stakeholders updated on their investments even before the project is completed.

See also 7.08 - Maturity of Life Cycle Products at Milestone Reviews, 7.09 - Entrance and Exit Criteria

3.2 How To Select Milestones

When creating your own project management schedule, it can be challenging to determine what to set as milestones. Too few milestones may not provide you with enough project oversight, but too many milestones clutter up your reports and reduce their value.

Achieve the right balance, it’s not as simple as just listing every deliverable due date. Project milestones need to have a clear and consistent meaning, both throughout and across projects. For instance, if “submission of initial draft” is a milestone on one project, it should be a milestone on all similar projects. This creates consistency for reporting, which enables stakeholders to better understand the overall progress of all projects.

If you’re not sure what should and should not be a milestone, the following questions may help:

  • Is it an activity that requires time, or is it an end product?
  • Will it impact the final deadline?
  • Is it an important moment in the project?
  • Will it be a strong indicator of overall project progress?
  • Does it need to be reviewed and approved by stakeholders?
  • Is it an event that impacts the project?

Key deliverables owed to you by outside parties are also useful deadlines. For instance, imagine that you require a subcontractor to complete some sub-assembly before your team can begin production. In this scenario, the due date for the subcontractor should be a milestone. Another example is if you have a critical material delivery expected to arrive in six weeks. That arrival date may need to be a milestone in your schedule.

3.3 How To Use Milestones In Your Project Management Software

One of the best ways to organize your project effectively is to interconnect the tasks with the project goals. That makes tasks meaningful for your team and motivates people to get things done faster.

For example, if you’re launching a new software update, major milestones may include:

Finalize software design
Finalize software production
Complete software testing
You can then organize the associated tasks by the project milestones. This allows everyone to see which concrete steps need to be achieved to reach each goal, and it gives team members a better understanding of the impact of their individual tasks.

Milestones can also be used to define check-in points throughout the project so that everyone is clear about what progress looks like, what the expectations are, and when they’ll be measured. By inputting these checkpoints into the schedule and managing them, you enable the team to stay in sync with overall project goals and outcomes.

3.4 Tasks Associated With A Milestone Review

  1. The project Manager calculates milestone progress and prepares a milestone review report using the milestones review report template.
  2. The project manager sends the meeting request to the relevant stakeholders along with the meeting agenda and milestone review report as per plan.
  3. The project manager conducts milestones review meetings according to the meeting agenda.
  4. All agenda items are discussed, and issues, improvement suggestions, lessons learned, and action items are noted and logged in the issue management system.
  5. The project manager sends a milestone review report to the project and customer contact person.
  6. The project manager assigns the action items to appropriate stakeholders and obtains their commitment.
  7. The project manager tracks the action items to closure.

See also SWE-018 - Software Activities Review

3.5 Acquired Software Development

For acquired software development, milestone reviews are incorporated into the contract because the contract is the binding document for contractor performance and deliverables. Regardless of whether the development is in-house or contracted, the development agreement needs to contain, among other key elements, surveillance activities including monitoring activities, reviews, audits, decision points, meetings, known contract milestones, etc.

See also Topic 8.06 - IV&V Surveillance

Other items related to milestone reviews to include in the development agreement are:

  • Review periods for deliverables.
  • The period for making corrections to resolve findings.
  • Formal reviews, such as those found in NPR 7123.1 041, NPR 7120.5 083, NPR 7120.7 264 (IT and Institutional Infrastructure), and NPR 7120.8 269(Research and Technology).
  • Technical reviews.
  • Progress reviews.
  • Acceptance reviews.

Consult Center guidance on the following topics as Center guidance may provide insights into reviews and review topics to be considered for inclusion in the Statement of Work (SOW) and contract:

  • Acquisition.
  • Planning.
  • Scheduling.
  • Process Monitoring and Control (PMC).

The references in 7.03 - Acquisition Guidance may provide additional guidance on project milestone reviews and topics for consideration. Topic 7.09 - Entrance and Exit Criteria describes inputs, material that will be reviewed, and outputs for each life cycle milestone review which may be useful as input to the development of checklists for these reviews. Consult the NPR 7120 family of requirements documents for definitions of milestones and approaches for different project types.

Keep in mind that reviews not included in the contract may be difficult to require of the contractor, so it is important to ensure the SOW and other contract elements are reviewed by the proper project management and/or technical authority for completeness.

3.6 Additional Guidance

Additional guidance related to this requirement may be found in the following materials in this Handbook:

3.7 Center Process Asset Libraries

SPAN - Software Processes Across NASA
SPAN contains links to Center managed Process Asset Libraries. Consult these Process Asset Libraries (PALs) for Center-specific guidance including processes, forms, checklists, training, and templates related to Software Development. See SPAN in the Software Engineering Community of NEN. Available to NASA only. https://nen.nasa.gov/web/software/wiki  197

See the following link(s) in SPAN for process assets from contributing Centers (NASA Only). 

4. Small Projects

Project review plans and milestones are covered in NPR 7120.5 083, NPR 7120.7 264, NPR 7120.8 269, and NPR 7123.1 041. Projects are to ensure that software components are a part of specific project milestone reviews. Small projects need to determine a review process that meets the project requirements and contains adequate content to provide the greatest insights into progress toward the project's technical goals and the technical direction of the project.

5. Resources

5.1 References


5.2 Tools

Tools to aid in compliance with this SWE, if any, may be found in the Tools Library in the NASA Engineering Network (NEN). 

NASA users find this in the Tools Library in the Software Processes Across NASA (SPAN) site of the Software Engineering Community in NEN. 

The list is informational only and does not represent an “approved tool list”, nor does it represent an endorsement of any particular tool.  The purpose is to provide examples of tools being used across the Agency and to help projects and centers decide what tools to consider.

6. Lessons Learned

6.1 NASA Lessons Learned

A documented lesson from the NASA Lessons Learned database notes the following:

  • Acquisition and Oversight of Contracted Software Development (1999). Lesson Number 0921 528: Tailorable acquisition management and oversight processes for NASA contracted software development are essential to ensure that customers receive a quality product. A documented lesson from the NASA Lessons Learned database includes a cause of the loss of a mission "the lack of a controlled and effective process for acquisition of contractor-developed, mission-critical software." In this particular case, the quality of the contractor's product was not monitored as it would have been if the proper milestones for reviewing and auditing contractor progress were in place.

6.2 Other Lessons Learned

No other Lessons Learned have currently been identified for this requirement.

7. Software Assurance

SWE-037 - Software Milestones
3.1.7 The project manager shall define and document the milestones at which the software developer(s) progress will be reviewed and audited. 

7.1 Tasking for Software Assurance

From NASA-STD-8739.8B

1. Confirm that milestones for reviewing and auditing software developer progress are defined and documented.

2. Participate in project milestones reviews.

7.2 Software Assurance Products

  •  Software Assurance Status Reports
  • SA audit schedule is consistent with the project schedule.
  • List of any issues//risks identified with schedule or developer progress.


    Objective Evidence

    • Milestone review plans.
    • Definition for software products life cycle entrance and exit criteria.
    • Evidence of SA participation in milestone reviews.

    Objective evidence is an unbiased, documented fact showing that an activity was confirmed or performed by the software assurance/safety person(s). The evidence for confirmation of the activity can take any number of different forms, depending on the activity in the task. Examples are:

    • Observations, findings, issues, risks found by the SA/safety person and may be expressed in an audit or checklist record, email, memo or entry into a tracking system (e.g. Risk Log).
    • Meeting minutes with attendance lists or SA meeting notes or assessments of the activities and recorded in the project repository.
    • Status report, email or memo containing statements that confirmation has been performed with date (a checklist of confirmations could be used to record when each confirmation has been done!).
    • Signatures on SA reviewed or witnessed products or activities, or
    • Status report, email or memo containing a short summary of information gained by performing the activity. Some examples of using a “short summary” as objective evidence of a confirmation are:
      • To confirm that: “IV&V Program Execution exists”, the summary might be: IV&V Plan is in draft state. It is expected to be complete by (some date).
      • To confirm that: “Traceability between software requirements and hazards with SW contributions exists”, the summary might be x% of the hazards with software contributions are traced to the requirements.
    • The specific products listed in the Introduction of 8.16 are also objective evidence as well as the examples listed above.

SA Products for Milestone Reviews

ReviewSoftware Assurance Product for Review
All Reviews
  • SA's general assessment of the status and quality of the software and safety activities
  • Any schedule/progress, quality, or safety concerns
  • High-level results of any audits, peer reviews, assessments, or analyses
Additional Software Assurance /Safety Products for Milestone Reviews
System Requirements Review (SRR)
  • Independent SA Software Classification or Concurrence on Software Engineering's Software Classification
  • SA Safety-Criticality Determination
  • Preliminary Hazard Analysis
  • Preliminary SA Plan, SA schedule, and SA processes
Software Requirements Review (SwRR)
  • Software Assurance/Quality Plan
  • Preliminary Safety Plan
  • SA Requirements Assessment, including any issues with bidirectional traceability
Mission Definition Review (MDR)
  • Updated Software Assurance/Safety Plan
  • Preliminary SA Safety Analysis
  • SA status, including results from any assessments, audits, peer reviews (See all reviews above)
System Definition Review (SDR)
  • Software Safety Analysis
  • Software Assurance/Safety Plan (for baselining)
  • SA status, including results from any assessments, audits, peer reviews (See all reviews above)
Preliminary Design Review (PDR)
  • Software Safety Analysis, including FTA, FMEA (if done)
  • SA Compliance Matrix
  • SA status, including results from any assessments, audits, peer reviews (See all reviews above)
Critical Design Review (CDR)
  • Hazard Analysis
  • SA Safety Plan with verifications
  • SA assessment of design
  • SA analysis of software progress, metrics
  • SA status of safety and assurance activities, including results from any assessments, audits, peer reviews
Production Readiness Review (PRR)
  • SA status (See all reviews above)
System Integration Review (SIR)
  • SA status (as in all reviews), including any concerns with safety or the integration process
  • SA analysis of static code analysis results
  • SA assessment of integration plans, procedures, or handling of safety requirements
  • Status of system discrepancies, system test results
Test Readiness Review (TRR)
  • SA assessment of test readiness, test procedures, witnessing plans (or SA participation plans)
  • SA analysis of software metrics, software progress
  • SA results of any code assessments, audits, peer reviews
  • SA results of Version Description Document (VDD) and Functional Configuration Audits (FCA)
System Acceptance Review (SAR)
  • SA Status
  • Results of configuration audits: Functional Configuration Audits( FCAs) and Physical Configuration Audits (PCAs)
  • Status of documentation, previous testing, including assessment of metrics on non-conformance/Problem Reports (PRs)/Discrepancy Reports (DRs)
Operational Readiness Review (ORR)
  • Physical Configuration Audit (PCA) results
  • SA assessment of V&V status and documentation
  • SA status on safety and security activities
  • SA assessment of readiness for operations and maintenance
  • Results of any other audits, assessments, and metrics analysis
Flight Readiness Review (FRR)
  • SA assessment of readiness for flight
  • Any SA concerns with safety or security
  • SA sign-offs on certification packages

See also Topic 8.05 - SW Failure Modes and Effects Analysis, 8.07 - Software Fault Tree Analysis, 8.12 - Basics of Software Auditing

7.3 Metrics

  • # of Non-Conformances from reviews (Open vs. Closed; # of days Open)

7.4 Guidance

List of tasks for the software development required for the project’s software developers. - look at the software development schedule milestones (see 7.08 - Maturity of Life Cycle Products at Milestone Reviews, software products required, and software processes used. Related to the SA tasks on SWE-036 - Software Process Determination

7.5 Additional Guidance

Additional guidance related to this requirement may be found in the following materials in this Handbook:

  • No labels

0 Comments