5. Resources

5.1 References

5.2 Tools

6. Lessons Learned

6.1 NASA Lessons Learned

No Lessons Learned have currently been identified for this requirement.

6.2 Other Lessons Learned

• Throughout hundreds of inspections and analyses of their results, the Jet Propulsion Laboratory (JPL) has identified key lessons learned which lead to more effective inspections

  Swerefn
  refnum 235

  , including:
  • Inspections are carried out by peers representing the areas of the life cycle affected by the material being inspected. Everyone participating should have a vested interest in the work product.
  • Management is not present during inspections.
  • Checklists of questions are used to define the task and to stimulate defect findings.

7. Software Assurance

7.1 Tasking for Software Assurance

7.2 Software Assurance Products

• Peer Review Process Audit Report (SA audit results and findings on software peer-review process).

  
  

  Note
  title Objective Evidence
  Peer review metrics, reports, data, or findings List of participates in the software peer reviews Defect or problem reporting tracking data Software assurance audit reports on the peer-review process

  Expand
  title Definition of objective evidence
  Include Page Definition of Objective Evidence Definition of Objective Evidence

7.3 Metrics

• # of software process Non-Conformances by life cycle phase over time
• Preparation time each audit participant spent preparing for the audit
• Time required to close peer review audit Non-Conformances
• Trends on non-conformances from audits (Open, Closed, Life Cycle Phase)
• # of Peer Review Audits planned vs. # of Peer Review Audits performed
• # of audit Non-Conformances per peer review audit
• # of peer review Non-Conformances per work product vs. # of peer reviewers
• # of peer review participants vs. total # invited
• Preparation time each review participant spent preparing for the review
• Total # of peer review Non-Conformances (Open, Closed)
• # of Non-Conformances identified by software assurance during each peer review
• # of Non-Conformances from reviews (Open vs. Closed; # of days Open)
• # of process Non-Conformances (e.g., activities not performed) identified by SA vs. # accepted by the project
• Trends of # Open vs. # Closed over time
• # of Non-Conformances per audit (including findings from process and compliance audits, process maturity)
• # of Open vs. Closed Audit Non-Conformances over time
• Trends of # of Non-Conformances from audits over time (Include counts from process and standards audits and work product audits.)
• # of Compliance Audits planned vs. # of Compliance Audits performed

7.4 Guidance

Task 1: 

Confirm that the project has met the 4 conditions in the activities specified in the software requirement:

• Confirm that the project has prepared one or more checklists to evaluate the work product before the peer review.

The selection and preparation of the checklists to use during the review is part of the preparation for the review that needs to be completed, along with other practical considerations such as determining who should attend the review, when and where the review should be held, reserving a room, and distributing the review announcement and material to be reviewed.

Each review should have specific checklists, depending on the type of asset that is being reviewed, so different checklists would be used for reviewing software development plans, test plans, and procedures, configuration management plans, a design document, or a portion of the code. A checklist may also be written to capture a particular perspective of some roles of the team. For example, there may be checklist(s) from a perspective of a person on the requirements development team, the design team, the coding team, the test team, the operations team, or software assurance.

Checklists should be used during the review for guidance on typical types of defects to be found in the type of product being inspected. In addition, the product being inspected is checked against higher-level work products, standards, and interface documents to assure compliance and correctness.

There are many inspection checklists available that can be used as a starting point for different kinds of reviews. Check with your Center process asset team or look in SPAN for example. 

• The next item to confirm is that the entrance and exit criteria have been established.  Before the peer review, software assurance should confirm that the entrance criteria have been established and before the close-out of the review, software assurance should confirm that the exit criteria have been met. For the entrance criteria, check on the following:
  • Has the leader selected the material to be peer-reviewed at this review, and sent it out to the participants, along with any necessary background material?
  • Has the leader reserved a room, prepared checklists, selected participants with varying roles in the project?
  • Have the participants reviewed the materials in advance of the review?
• For the exit criteria, consider the following:
  • Have all the identified issues and defects been recorded and assigned to further investigation and resolution?
  • Has a priority been assigned to the issues and defects?
  • Have the selected metrics been collected and recorded?
  • Needs a re-inspection been established?
  • Have all the issues and problems been closed out? Software assurance should be tracking these issues and verifying they are closed out.

• The third activity software assurance needs to confirm is that the peer review items have been resolved before the closure of the review. Software assurance will do this by tracking the item independently and verifying that items are closed.

• The final activity that software assurance will confirm is that the selection of the participants has been made by the leader. Several considerations should be considered when selecting review team members.

• • Often peer reviews are more effective if the team size is limited to 5 to 9 people.
  • Team members must have technical knowledge about the project and be familiar with the asset being reviewed. If this is not the case, it is recommended that the leader give the team an overview before the review.
  • Team members should be selected to provide different perspectives on the product being reviewed. So, team members might be selected from the following (depending on the product being reviewed): requirements developers, designers, coders, test team members, software assurance, or knowledgeable peers from a similar project.
  • Usually, it is best to avoid including the manager of the product being reviewed as a team member. The primary purpose of the peer reviews is to find errors and there might be some hesitation to mention all the defects for fear of a poor performance rating.
  • Specific roles may be assigned to the participants to make sure the review is efficient. For example, someone may be assigned to “read” or verbally walk through the product, while someone else is assigned to record the issues and defects.

Task 2:

The other SA assigned task is to confirm that all the actions from the review have been closed out. The software assurance personnel will do this by tracking the action items and verifying they have been closed. Before the final close-out of the review, the software assurance personnel should meet with the review lead to verify no outstanding issues remain.

Task 3:

Audit the peer review process at least once every year. Every task that involves performing an audit should also clarify that all audit findings are promptly shared with the project and that improvements to the peer review process will be addressed in the handbook guidance.