This requirement calls out four important best practices that are associated with effective inspections:
a. Using a checklist supports the software peer review or software inspection team members by giving them a memory aid regarding what quality aspects they are responsible for in the document under review. The checklists provide a concrete way for the inspection to improve over time. Defect types that are seen to continually slip through peer reviews or software inspections are added to the checklist so that future teams are aware that they are important to look for. Checklist items that no longer lead to defects being found are candidates for deletion. If kept up to date in this way, checklists provide a timely and efficient list of the types of issues on which review time should be spent.
Using a formal reading technique such as perspective-based reading helps ensure that the viewpoints of the various customers and stakeholders of the product under review are represented. Peer review or inspection team members take on the roles to represent the different points of view. “The goal ... is to provide operational scenarios where members of a review team read a document from a particular perspective, e.g., tester, developer, the user. The combination of different perspectives provides better coverage of the document, i.e., uncovers a wider range of defects, than the same number of readers using their usual technique.”
b. Readiness and completion criteria are used to ensure that peer review or software inspection time is being spent effectively and that confidence can be had in the outcome. Readiness criteria are satisfied before an inspection is able to begin. They represent the minimal set of quality characteristics that are to be satisfied before it is worthwhile to have a team of subject matter experts spends significant time understanding, assessing, and discussing the product under review or inspection. Readiness criteria also indicate the preparedness of the peer review or software inspection team to conduct the review or inspection. Readiness criteria may specify standards and guidelines to be adhered to; set project-specific criteria like the level of detail or a particular policy to be followed, and may require the use of automated tools (like static analysis tools or traceability tools). Completion criteria represent a set of measurable activities that are to be completed at the end of the inspection so that statements can be made with confidence regarding the outcome. For example, completion criteria may require that all process steps have been completed and documented; metrics have been collected; or that all major defects have been completed and approved.
Table G-19 below is from the NASA System Engineering Processes and Requirements, NPR 7123.1, and shows the entrance criteria and success criteria for a peer review activity.
Table G-19 - Peer Review Entrance and Success Criteria
- The product to be reviewed (e.g., document, process, model, design details) has been identified and made available to the review team.
- Peer reviewers independent from the project have been selected for their technical background related to the product being reviewed.
- A preliminary agenda, success criteria, and instructions to the review team have been agreed to by the technical team and project manager.
- Rules have been established to ensure consistency among the team members involved in the peer-review process.
- *Spectrum (radio frequency) considerations addressed.
- Peer review has thoroughly evaluated the technical integrity and quality of the product.
- Any defects have been identified and characterized.
- The results of the peer review are communicated to the appropriate project personnel.
- Spectrum-related aspects have been concurred to by the responsible Center spectrum manager.
*Required per NPD 2570.5.
c. Action items are required to be tracked through completion so that it is assured that the inspection has a positive impact on software quality. Due to time pressures, teams who identify significant numbers of defects in inspection and then do not take the time to resolve them, are wasting effort. Tracking the action items ensures that such an outcome is avoided. In addition to the impact on software quality, this best practice also aims at keeping the morale of inspection teams high. Nothing is more demoralizing for a team than investing significant time in identifying and reporting software defects if they are never fixed afterward.
d. Effective peer reviews or software inspections begin with a planning phase in which plans are made regarding the scope of the document under review, the time available, and other key parameters. One of the most important issues to address in this step is to analyze which perspectives of stakeholders are needed to ensure that all quality aspects can be adequately addressed in an inspection. Taking the time to apply a rigorous inspection process will not automatically yield an effective outcome if the actual engineering knowledge and expertise are never brought to bear on analyzing the document.
NASA-STD-8739.9, Software Formal Inspections Standard suggests several best practices related to the use of checklists. They recommend that:
- Each team member uses a checklist or similar work aid available, with items relevant to the perspective each is representing.
- Checklists are included as an input to any inspection.
- Inspectors use the given checklists during their preparation.
The Standard offers detailed suggestions as to what types of quality aspects need to be covered by checklists in a variety of different circumstances.
The Standard also suggests that the perspectives of key stakeholders be represented on the inspection team. Recommended practices include:
- Choose inspectors in consultation with the author.
- Moderator ensures that objectivity in the selection is maintained.
- One individual may represent multiple perspectives.
- Inspectors representing key perspectives must be present and prepared for each relevant stage of the inspection process.
Best practices related to the establishment of readiness and completion criteria include:
- Entrance and exit criteria are specified as part of the inspection procedure, and provide several examples of criteria found useful on NASA teams.
- During the inspection planning, the work product under inspection is evaluated against the entrance criteria before the inspection can begin.
- The project manager defines the criteria to be used to determine if an inspection ends by passing the document under review, or requiring a re-inspection.
- To ensure that close-out activities are undertaken, at the end of any inspection meeting, the moderator:
- Determines based on the outcome of inspections, using the criteria previously defined by the project manager, if a re-inspection will be needed.
- Compiles, as the outcome of an inspection meeting:
- A list of classified anomalies or defects identified from the inspections.
- A list of change requests or discrepancy reports for defects found in work products of the previous development phase that have been put under configuration management (CM).
- The inspected work product marked with clerical defects.
- Ensures that authors of the work product inspected receive the list of classified anomalies or defects.
Best practices related to tracking actions identified in the reviews until they are resolved to include:
- Action items and defects discussed during the inspection meeting are compiled and tracked starting at that time.
- The author's fixes to defects discovered during an inspection are verified before the end of that inspection.
- Each project defines a method for documenting, tracking, and measuring such action items.
Best practices related to the identification of required participants include:
- The team consists of a minimum of three inspectors, reflecting that diverse viewpoints and objectivity are required to be brought to bear during an inspection.
- The inspection team members are based on an analysis of the key stakeholders in the document under inspection.
The Fraunhofer Center in Maryland maintains a public website that collects checklists found from NASA and other contexts, which can be applied to the types of work products mentioned in these requirements.
NASA users should consult Center Process Asset Libraries (PALs) for Center-specific guidance and resources, such as templates, related to peer reviews and inspections.