This version of SWEHB is associated with NPR 7150.2B. Click for the latest version of the SWEHB based on NPR7150.2D

SWE-088 - Software Peer Reviews and Inspections - Checklist Criteria and Tracking

1. Requirements

5.3.3 The project manager shall, for each planned software peer review or software inspection:

a. Use a checklist or formal reading technique (e.g., perspective based reading) to evaluate the work products.

b. Use established readiness and completion criteria.

c. Track actions identified in the reviews until they are resolved.

d. Identify required participants.

1.1 Notes

NPR 7150.2, NASA Software Engineering Requirements, does not include any notes for this requirement.

1.2 Applicability Across Classes

If Class D software is safety critical, this requirement applies to the safety-critical aspects of the software.

Classes F and G are labeled with "X (not OTS)." This means that this requirement does not apply to off-the-shelf software for these classes.























Key:    - Applicable | - Not Applicable

2. Rationale

Each part listed in the requirement is needed in order to conduct an effective peer review or inspection.  Peer reviews and inspections contribute to product and process quality, risk reduction, confirmation of approach, defect identification, and product improvements.

3. Guidance

This requirement calls out four important best practices that are associated with effective inspections:

a. Using a checklist supports the software peer review or software inspection team members by giving them a memory aid regarding what quality aspects they are responsible for in the document under review. The checklists provide a concrete way for the inspection to improve over time. Defect types that are seen to continually slip through peer reviews or software inspections are added to the checklist so that future teams are aware that they are important to look for. Checklist items that no longer lead to defects being found are candidates for deletion. If kept up to date in this way, checklists provide a timely and efficient list of the types of issues on which review time should be spent.

Using a formal reading technique such as perspective-based reading helps ensure that the viewpoints of the various customers and stakeholders of the product under review are represented. Peer review or inspection team members take on the roles to represent the different points of views. “The goal ... is to provide operational scenarios where members of a review team read a document from a particular perspective, e.g., tester, developer, user.  ...the combination of different perspectives provides better coverage of the document, i.e., uncovers a wider range of defects, than the same number of readers using their usual technique.” 474

b. Readiness and completion criteria are used to ensure that peer review or software inspection time is being spent effectively and that confidence can be had in the outcome. Readiness criteria are satisfied before an inspection is able to begin. They represent the minimal set of quality characteristics that are to be satisfied before it is worthwhile to have a team of subject matter experts spend significant time on understanding, assessing, and discussing the product under review or inspection. Readiness criteria also indicate the preparedness of the peer review or software inspection team to conduct the review or inspection. Readiness criteria may specify standards and guidelines to be adhered to; set project-specific criteria like the level of detail or a particular policy to be followed; and may require the use of automated tools (like static analysis tools or traceability tools). Completion criteria represent a set of measurable activities that are to be completed at the end of an inspection, so that statements can be made with confidence regarding the outcome. For example, completion criteria may require that all process steps have been completed and documented; metrics have been collected; or that all major defects have been completed and approved.

The table below is from the NASA System Engineering Processes and Requirements, NPR 7123.1, and shows the entrance criteria and success criteria for a peer review activity.

c. Action items are required to be tracked through completion so that it is assured that the inspection has a positive impact on software quality. Due to time pressures, teams who identify significant numbers of defects in an inspection and then do not take the time to resolve them, are wasting effort. Tracking the action items ensures that such an outcome is avoided. In addition to the impact on software quality, this best practice also aims at keeping the morale of inspection teams high. Nothing is more de-moralizing for a team than investing significant time in identifying and reporting software defects, if they are never fixed afterwards.

d. Effective peer reviews or software inspections begin with a planning phase in which plans are made regarding the scope of the document under review, the time available, and other key parameters. One of the most important issues to address in this step is to analyze which perspectives or stakeholders are needed to ensure that all quality aspects can be adequately addressed in an inspection. Taking the time to apply a rigorous inspection process will not automatically yield an effective outcome if the actual engineering knowledge and expertise is never brought to bear on analyzing the document.

NASA-STD-8739.9, Software Formal Inspections Standard suggests several best practices related to the use of checklists. They recommend that:

  • Each team member use a checklist or similar work aid available, with items relevant to the perspective each is representing.
  • Checklists are included as an input to any inspection.
  • Inspectors use the given checklists during their preparation. 277

The Standard offers detailed suggestions as to what types of quality aspects need to be covered by checklists in a variety of different circumstances.

The Standard also suggests that the perspectives of key stakeholders be represented on the inspection team.  Recommended practices include:

  • Choose inspectors in consultation with the author.
  • Moderator ensures that objectivity in the selection is maintained.
  • One individual may represent multiple perspectives.
  • Inspectors representing key perspectives must be present and prepared for each relevant stage of the inspection process. 277

Best practices related to the establishment of readiness and completion criteria include:

  • Entrance and exit criteria are specified as part of the inspection procedure, and provide several examples of criteria found useful on NASA teams.
  • During the inspection planning, the work product under inspection is evaluated against the entrance criteria before the inspection can begin.
  • The project manager defines the criteria to be used to determine if an inspection ends by passing the document under review, or requiring a re-inspection.
  • To ensure that close-out activities are undertaken, at the end of any inspection meeting, the moderator:
    • Determines based on the outcome of inspections, using the criteria previously defined by the project manager, if a re-inspection will be needed.
    • Compiles, as the outcome of an inspection meeting:
      • A list of classified anomalies or defects identified from the inspections.
      • A list of change requests or discrepancy reports for defects found in work products of the previous development phase that have been put under configuration management (CM).
      • The inspected work product marked with clerical defects.
    • Ensures that authors of the work product inspected receive the list of classified anomalies or defects.

Best practices related to tracking actions identified in the reviews until they are resolved include:

  • Action items and defects discussed during the inspection meeting are compiled and tracked starting at that time.
  • The author's fixes to defects discovered during an inspection are verified before the end of that inspection.
  • Each project defines a method for documenting, tracking, and measuring such action items.

Best practices related to the identification of required participants include:

  • The team consists of a minimum of three inspectors, reflecting that diverse viewpoints and objectivity are required to be brought to bear during an inspection.
  • The inspection team members are based on an analysis of the key stakeholders in the document under inspection.

The Fraunhofer Center 421 in Maryland maintains a public website that collects checklists found from NASA and other contexts, which can be applied to the types of work products mentioned in these requirements.

NASA users should consult Center Process Asset Libraries (PALs) for Center-specific guidance and resources, such as templates, related to peer reviews and inspections.

4. Small Projects

Checklists for various types of inspections can be found at the Fraunhofer Center website 421. Various inspection tools can be used to reduce the effort of tracking the information associated with inspections. See the "Tools" section of the Resources tab for a list of tools.

5. Resources

5.1 Tools

Tools to aid in compliance with this SWE, if any, may be found in the Tools Library in the NASA Engineering Network (NEN).

NASA users find this in the Tools Library in the Software Processes Across NASA (SPAN) site of the Software Engineering Community in NEN.

The list is informational only and does not represent an “approved tool list”, nor does it represent an endorsement of any particular tool. The purpose is to provide examples of tools being used across the Agency and to help projects and centers decide what tools to consider.

6. Lessons Learned

Over the course of hundreds of inspections and analysis of their results, the Jet Propulsion Laboratory (JPL) has identified key lessons learned which lead to more effective inspections, including:

  • Inspections are carried out by peers representing the areas of the life cycle affected by the material being inspected. Everyone participating should have a vested interest in the work product.
  • Management is not present during inspections.
  • Checklists of questions are used to define the task and to stimulate defect finding 235.

  • No labels