- 1. The Requirement
- 2. Rationale
- 3. Guidance
- 4. Small Projects
- 5. Resources
- 6. Lessons Learned
- 7. Software Assurance
5.3.3 The project manager shall, for each planned software peer review or software inspection:
- Use a checklist or formal reading technique (e.g., perspective based reading) to evaluate the work products.
- Use established readiness and completion criteria.
- Track actions identified in the reviews until they are resolved.
- Identify the required participants.
NPR 7150.2, NASA Software Engineering Requirements, does not include any notes for this requirement.
Click here to view the history of this requirement: SWE-088 History
1.2 Applicability Across Classes
Key: - Applicable | - Not Applicable
A & B = Always Safety Critical; C & D = Sometimes Safety Critical; E - F = Never Safety Critical.
Checklists, criteria, and tracking of actions and participants are needed in order to conduct an effective peer review or inspection. Peer reviews and inspections contribute to product and process quality, risk reduction, confirmation of approach, defect identification, and product improvements.
This requirement calls out four important best practices that are associated with effective inspections:
a. Using a checklist supports the software peer review or software inspection team members by giving them a memory aid regarding what quality aspects they are responsible for in the document under review. The checklists provide a concrete way for the inspection to improve over time. Defect types that are seen to continually slip through peer reviews or software inspections are added to the checklist so that future teams are aware that they are important to look for. Checklist items that no longer lead to defects being found are candidates for deletion. If kept up to date in this way, checklists provide a timely and efficient list of the types of issues on which review time should be spent.
Using a formal reading technique such as perspective-based reading helps ensure that the viewpoints of the various customers and stakeholders of the product under review are represented. Peer review or inspection team members take on the roles to represent the different points of view. “The goal ... is to provide operational scenarios where members of a review team read a document from a particular perspective, e.g., tester, developer, the user. The combination of different perspectives provides better coverage of the document, i.e., uncovers a wider range of defects, than the same number of readers using their usual technique.” 474
b. Readiness and completion criteria are used to ensure that peer review or software inspection time is being spent effectively and that confidence can be had in the outcome. Readiness criteria are satisfied before an inspection is able to begin. They represent the minimal set of quality characteristics that are to be satisfied before it is worthwhile to have a team of subject matter experts spends significant time understanding, assessing, and discussing the product under review or inspection. Readiness criteria also indicate the preparedness of the peer review or software inspection team to conduct the review or inspection. Readiness criteria may specify standards and guidelines to be adhered to; set project-specific criteria like the level of detail or a particular policy to be followed, and may require the use of automated tools (like static analysis tools or traceability tools). Completion criteria represent a set of measurable activities that are to be completed at the end of the inspection so that statements can be made with confidence regarding the outcome. For example, completion criteria may require that all process steps have been completed and documented; metrics have been collected; or that all major defects have been completed and approved.
Table G-19 below is from the NASA System Engineering Processes and Requirements, NPR 7123.1, and shows the entrance criteria and success criteria for a peer review activity.
Table G-19 - Peer Review Entrance and Success Criteria
*Required per NPD 2570.5.
c. Action items are required to be tracked through completion so that it is assured that the inspection has a positive impact on software quality. Due to time pressures, teams who identify significant numbers of defects in inspection and then do not take the time to resolve them, are wasting effort. Tracking the action items ensures that such an outcome is avoided. In addition to the impact on software quality, this best practice also aims at keeping the morale of inspection teams high. Nothing is more demoralizing for a team than investing significant time in identifying and reporting software defects if they are never fixed afterward.
d. Effective peer reviews or software inspections begin with a planning phase in which plans are made regarding the scope of the document under review, the time available, and other key parameters. One of the most important issues to address in this step is to analyze which perspectives of stakeholders are needed to ensure that all quality aspects can be adequately addressed in an inspection. Taking the time to apply a rigorous inspection process will not automatically yield an effective outcome if the actual engineering knowledge and expertise are never brought to bear on analyzing the document.
NASA-STD-8739.9, Software Formal Inspections Standard suggests several best practices related to the use of checklists. They recommend that:
- Each team member uses a checklist or similar work aid available, with items relevant to the perspective each is representing.
- Checklists are included as an input to any inspection.
- Inspectors use the given checklists during their preparation. 277
The Standard offers detailed suggestions as to what types of quality aspects need to be covered by checklists in a variety of different circumstances.
The Standard also suggests that the perspectives of key stakeholders be represented on the inspection team. Recommended practices include:
- Choose inspectors in consultation with the author.
- Moderator ensures that objectivity in the selection is maintained.
- One individual may represent multiple perspectives.
- Inspectors representing key perspectives must be present and prepared for each relevant stage of the inspection process. 277
Best practices related to the establishment of readiness and completion criteria include:
- Entrance and exit criteria are specified as part of the inspection procedure, and provide several examples of criteria found useful on NASA teams.
- During the inspection planning, the work product under inspection is evaluated against the entrance criteria before the inspection can begin.
- The project manager defines the criteria to be used to determine if an inspection ends by passing the document under review, or requiring a re-inspection.
- To ensure that close-out activities are undertaken, at the end of any inspection meeting, the moderator:
- Determines based on the outcome of inspections, using the criteria previously defined by the project manager, if a re-inspection will be needed.
- Compiles, as the outcome of an inspection meeting:
- A list of classified anomalies or defects identified from the inspections.
- A list of change requests or discrepancy reports for defects found in work products of the previous development phase that have been put under configuration management (CM).
- The inspected work product marked with clerical defects.
- Ensures that authors of the work product inspected receive the list of classified anomalies or defects.
Best practices related to tracking actions identified in the reviews until they are resolved to include:
- Action items and defects discussed during the inspection meeting are compiled and tracked starting at that time.
- The author's fixes to defects discovered during an inspection are verified before the end of that inspection.
- Each project defines a method for documenting, tracking, and measuring such action items.
Best practices related to the identification of required participants include:
- The team consists of a minimum of three inspectors, reflecting that diverse viewpoints and objectivity are required to be brought to bear during an inspection.
- The inspection team members are based on an analysis of the key stakeholders in the document under inspection.
The Fraunhofer Center 421 in Maryland maintains a public website that collects checklists found from NASA and other contexts, which can be applied to the types of work products mentioned in these requirements.
NASA users should consult Center Process Asset Libraries (PALs) for Center-specific guidance and resources, such as templates, related to peer reviews and inspections.
4. Small Projects
Checklists for various types of inspections can be found at the Fraunhofer Center website 421. Various inspection tools can be used to reduce the effort of tracking the information associated with inspections. See the "Tools" section of the Resources tab for a list of tools.
- Dr. Forrest Shull, at the Fraunhofer Center - Maryland, is a SME on the subject of inspection checklists. He has worked with NASA teams on inspections for several years and can be reached at firstname.lastname@example.org.A variety of articles from Dr. Forrest Shull is available on this cite.
6. Lessons Learned
6.1 NASA Lessons Learned
No Lessons Learned have currently been identified for this requirement.
6.2 Other Lessons Learned
- Over the course of hundreds of inspections and analysis of their results, the Jet Propulsion Laboratory (JPL) has identified key lessons learned which lead to more effective inspections 235, including:
- Inspections are carried out by peers representing the areas of the life cycle affected by the material being inspected. Everyone participating should have a vested interest in the work product.
- Management is not present during inspections.
- Checklists of questions are used to define the task and to stimulate defect finding.
7. Software Assurance
7.1 Tasking for Software Assurance
Confirm that the project meets the NPR 7150.2 criteria in "a" through "d" for each software peer review.
Confirm that the project resolves the actions identified from the software peer reviews.
Perform audits on the peer-review process.
7.2 Software Assurance Products
Peer Review Process Audit Report (SA audit results and findings on software peer review process).
Definition of objective evidence
- Evidence that confirms the NPR 7150.2 criteria in "a" through "d" for peer reviews have been met, including any risks and issues identified.
- Evidence of confirmation that the project resolves the actions identified from the software peer reviews.
Objective evidence is an unbiased, documented fact showing that an activity was confirmed or performed by the software assurance/safety person(s). The evidence for confirmation of the activity can take any number of different forms, depending on the activity in the task. Examples are:
- Observations, findings, issues, risks found by the SA/safety person and may be expressed in an audit or checklist record, email, memo or entry into a tracking system (e.g. Risk Log).
- Meeting minutes with attendance lists or SA meeting notes or assessments of the activities and recorded in the project repository.
- Status report, email or memo containing statements that confirmation has been performed with date (a checklist of confirmations could be used to record when each confirmation has been done!).
- Signatures on SA reviewed or witnessed products or activities, or
- Status report, email or memo containing Short summary of information gained by performing the activity. Some examples of using a “short summary” as objective evidence of a confirmation are:
- To confirm that: “IV&V Program Execution exists”, the summary might be: IV&V Plan is in draft state. It is expected to be complete by (some date).
- To confirm that: “Traceability between software requirements and hazards with SW contributions exists”, the summary might be x% of the hazards with software contributions are traced to the requirements.
- # of software process Non-Conformances by life-cycle phase over time
- Preparation time each audit participant spent preparing for audit
- Time required to close peer review audit Non-Conformances
- Trends on non-conformances from audits (Open, Closed, Life-cycle Phase)
- # of Peer Review Audits planned vs. # of Peer Review Audits performed
- # of audit Non-Conformances per peer review audit
- # of peer review Non-Conformances per work product vs. # of peer reviewers
- # of peer review participants vs. total # invited
- Preparation time each review participant spent preparing for the review
- Total # of peer review Non-Conformances (Open, Closed)
- # of Non-Conformances identified by software assurance during each peer review
- # of Non-Conformances from reviews (Open vs. Closed; # of days Open)
- # of process Non-Conformances (e.g., activities not performed) identified by SA vs. # accepted by the project
- Trends of # Open vs. # Closed over time
- # of Non-Conformances per audit (including findings from process and compliance audits, process maturity)
- # of Open vs. Closed Audit Non-Conformances over time
- Trends of # of Non-Conformances from audits over time (Include counts from process and standards audits and work product audits.)
- # of Compliance Audits planned vs. # of Compliance Audits performed
Confirm that the project has met the 4 conditions in the activities specified in the software requirement:
- Confirm that the project has prepared one or more checklists to evaluate the work product prior to the peer review.
The selection and preparation of the checklists to use during the review is part of the preparation for the review that needs to be completed, along with other practical considerations such as determining who should attend the review, when and where the review should be held, reserving a room, and distributing the review announcement and material to be reviewed.
Each review should have specific checklists, depending on the type of asset that is being reviewed, so different checklists would be used for reviewing software development plans, test plans, and procedures, configuration management plans, a design document, or a portion of the code. A checklist may also be written to capture a particular perspective of some roles of the team. For example, there may be checklist(s) from a perspective of a person on the requirements development team, the design team, the coding team, the test team, the operations team, or software assurance.
Checklists should be used during the review for guidance on typical types of defects to be found in the type of product being inspected. In addition, the product being inspected is checked against higher-level work products, standards, and interface documents to assure compliance and correctness.
There are many inspection checklists available that can be used as a starting point for different kinds of reviews. Check with your Center process asset team or look in SPAN for example.
- The next item to confirm is that the entrance and exit criteria have been established. Prior to the peer review, software assurance should confirm that the entrance criteria have been established and prior to the close-out of the review, software assurance should confirm that the exit criteria have been met. For the entrance criteria, check on the following:
- Has the leader selected the material to be peer-reviewed at this review, and sent it out to the participants, along with any necessary background material?
- Has the leader reserved a room, prepared checklists, selected participants with varying roles in the project?
- Have the participants reviewed the materials in advance of the review?
- For the exit criteria, consider the following:
- Have all the identified issues and defects been recorded and assigned to further investigation and resolution?
- Has a priority been assigned to the issues and defects?
- Have the selected metrics been collected and recorded?
- Has the need for a re-inspection been established?
- Have all the issues and problems been closed out? Software assurance should be tracking these issues and verify they are really closed out.
- The third activity software assurance needs to confirm is that the peer review items have been resolved prior to the closure of the review. Software assurance will do this by tracking the item independently and verifying that items are actually closed.
- The final activity that software assurance will confirm is that the selection of the participants has been made by the leader. There are a number of considerations that should be considered when selecting review team members.
- Often peer reviews are more effective if the team size is limited to 5 to 9 people.
- Team members must have technical knowledge about the project and be familiar with the asset being reviewed. If this is not the case, it is recommended that the leader give the team an overview prior to the review.
- Team members should be selected to provide different perspectives into the product being reviewed. So, team members might be selected from the following (depending on the product being reviewed): requirements developers, designers, coders, test team members, software assurance, or knowledgeable peers from a similar project.
- Usually, it is best to avoid including the manager of the product being reviewed as a team member. The primary purpose of the peer reviews is to find errors and there might be some hesitation to mention all the defects for fear of a poor performance rating.
- Specific roles may be assigned to the participants to make sure the review is efficient. For example, someone may be assigned to “read” or verbally walk-through the product, while someone else is assigned to record the issues and defects.
The other SA assigned task is to confirm that all the actions from the review have been closed out. The software assurance personnel will do this by tracking the action items and verifying they have been closed. Before the final close-out of the review, the software assurance personnel should meet with the review lead to verify no outstanding issues remain.
Audit the peer review process at least once every year. Every task that involves performing an audit should also clarify that all audit findings are promptly shared with the project and that improvements to the peer review process will be addressed in the handbook guidance.