Comment:
Migration of unmigrated content due to installation of a new plugin
Include Page
2B-Page Warning
2B-Page Warning
Set Data
hidden
true
name
reftab
1
Tabsetup
Introduction
11
SAR
Introduction
12
ORR
13
FRR
1
MCR
2
SRR
3
SwRR
4
MDR
5
SDR
6
PDR
7
CDR
8
PRR
9
SIR
10
TRR
Div
id
tabs-1
Entrance and Exit Criteria
Background
This guidance provides the maximum set of life cycle review entrance and exit criteria for software projects and should be tailored for the project class.
This guidance is a summarized collection of material from the following core documents: NPR 7123.1, Appendix G
Swerefn
refnum
041
; version D of NPR 7120.5 (since superseded by version E)
Swerefn
refnum
082
; and Center Procedures.
This guidance includes three types of information for each review:
Entrance criteria - Activities and products that are to be completed before the review can begin.
Materials for the Review - Items to be reviewed during review and used to confirm exit criteria; this information is typically available a couple of weeks prior to the review.
Exit criteria – Decisions and actions to be completed before the review is considered complete.
This guidance is focused on the responsibilities of the software engineering community throughout the project life cycle reviews. Therefore, the guidance includes reviews and products which are the primary responsibility of the software engineering community as well as software engineering community contributions to system activities and products, such as the Project Plan.
Note that different mission types (e.g., robotic vs. human) can have different life cycles and, therefore, different sets of life cycle reviews which apply.
This material considers a software project to be a system of systems as well as a single subsystem within the larger project. "System of systems" refers to a software project that includes software subsystems that perform functions allocated to them. Just as a project allocates requirements to hardware, software, external components, etc., software projects allocate software requirements to software subsystems.
Info
This material has been reviewed by the Software Working Group and the Office of the Chief Engineer.
Source of Content
Including the resources in the list below, information was pulled and consolidated based on repetition between Center Process Asset Libraries (PALs) and documents (Ames Research Center (ARC), Jet Propulsion Laboratory (JPL), Goddard Space Flight Center (GSFC), Marshall Space Flight Center (MSFC), Stennis Space Center (SSC)).
refstable-topic
Div
id
tabs-2
Mission Concept Review (MCR)
Info
The MCR affirms the mission need and examines the proposed mission's objectives and the concept for meeting those objectives. Key technologies are identified and assessed. It is an internal review that usually occurs at the cognizant system development organization. ROM (Rough Order of Magnitude) budget and schedules are presented. (NPR 7120.5
Preliminary risk assessment available, including technologies and associated risk management/mitigation strategies and options.
Conceptual test and evaluation strategy available.
A Mission Concept Review (MCR) agenda, success criteria, and charge to the board have been agreed to by the technical team, project manager, and review chair.
Preliminary technical plans available to achieve next phase.
Conceptual life cycle available.
Top level set of requirements are identified to meet the mission objectives.
The mission is feasible.
A solution has been identified that is technically feasible.
A rough cost estimate is within an acceptable cost range.
Draft cost and schedule estimates are available.
As developed by software (SW) developers and SW assurance personnel.
A technical search was done to identify existing assets/products that could satisfy the mission or parts of the mission
Software inputs/contributions provided for:
Preliminary Project Plan.
Preliminary Systems Engineering Management Plan (SEMP).
Development and analysis of alternative concepts (showing at least one feasible).
MCR Items Reviewed
Mission goals and objectives.
Analysis of alternative concepts.
Preliminary development approaches and acquisition plans.
Concept of operations.
Risk assessments.
Conceptual test and evaluation strategy.
Technical plans to achieve next phase.
Conceptual life cycle.
Preliminary requirements.
Draft cost and schedule estimates.
Conceptual system design.
MCR Exit/Success Criteria
Review panel agrees that:
Technical planning is sufficient to proceed to next phase.
Risk and mitigation strategies have been identified and are acceptable based on technical risk assessments.
Cost and schedule estimates are credible.
Mission goals and objectives are clearly defined and stated, unambiguous, and internally consistent.
Conceptual system design meets mission requirements, and the various system elements are compatible.
Technology dependencies are understood, and alternative strategies for achievement of requirements are understood.
As applicable, agreement is reached that:
Objectives are clearly understood and comprehensively defined.
Preliminary mission requirements are traceable to science objectives.
Operations concept clearly supports achievement of science objectives.
Div
id
tabs-3
System Requirements Review (SRR)
Info
The SRR examines the functional and performance requirements defined for the system and the preliminary Program or Project Plan and ensures that the requirements and the selected concept will satisfy the mission. (NPR 7120.5
Swerefn
refnum
082
)
If not performing a Software Requirements Review (SwRR), include SwRR criteria as part of SRR.
For software-only projects, the SwRR serves as the SRR.
Review of existing Lessons Learned (LL) from previous projects completed.
LL captured from software areas of the project (indicate the problem or success that generated the LL, what the LL was, and its applicability to future projects).
Confirmation exists that LL added to LL database.
SRR Items Reviewed
System-level requirements and preliminary allocation to next lower level-system (subsystems).
System-level software functionality description.
System-level interface requirements for software.
Concept of operations.
Mission requirements.
Preliminary Hazard Analysis (PHA).
Preliminary approach for how requirements will be verified and validated down to the subsystem level.
Risk and mitigation strategies.
Acquisition strategy.
SRR Exit/Success Criteria
Review panel agrees that:
Process for allocation and control of requirements throughout all levels is deemed sound; plan is defined to complete the definition activity within schedule constraints.
Requirements definition is complete with respect to top-level mission and science requirements; interfaces with external entities and between major internal elements are defined.
Preliminary allocation of system requirements to hardware, human, and software subsystems is defined.
Requirements allocation and flow down of key driving requirements are defined down to subsystems.
Preliminary approaches have been determined for how requirements will be verified and validated down to the subsystem level.
Major risks have been identified and technically assessed, and viable mitigation strategies defined.
Requirements and selected concept of operations will satisfy the mission.
System requirements, approved material solution, available product/process technology, and program resources are sufficient to proceed to the next life cycle phase.
Div
id
tabs-4
Software Requirements Review (SwRR)
Info
See definition of SRR.
If not performing a Software Requirements Review (SwRR), include SwRR criteria as part of SRR.
For software-only projects, the SwRR serves as the SRR.
Successful completion of the previous review (typically System Definition Review (SDR)) and responses made to all Requests for Actions (RFAs) and Review Item Discrepancies (RIDs).
A final Software Requirements Review (SwRR) agenda, success criteria, and charge to the board have been agreed to by the technical team, project manager, and review chair.
Technical products for this review made available to participants prior to SwRR.
Peer reviews completed: Software Management Plan (SMP), software requirements, verification and validation (V&V) planning.
Preliminary concept of operations available for review.
SwRR Entrance Criteria - Plans
Preliminary Software Development Plan (SDP)/Software Management Plan (SMP) updated for corresponding architectural design and test development activities
Preliminary approach and acquisition strategy.
Preliminary identification of personnel (quantity; assignment duration; required skills) or reference to document with this information.
Organizational responsibilities and interfaces.
Preliminary schedules exist for all software to be developed including dependencies with other disciplines within the project.
Preliminary cost estimate.
Milestones defined/revised.
Processes and metrics defined for program success.
Management methods and controls for design and development identified.
Programming languages (if known), security requirements, operational and support concepts identified
Training for project personnel identified.
Preliminary Software Quality Assurance Plan (SQAP) exists.
Preliminary Software Safety Plan exists (if have safety-critical software).
Preliminary software verification and validation (V&V) planning, including qualification requirements:
Overall software verification strategy.
Software development and test environments, including processors, operating systems, communications equipment, simulators and their fidelity.
Test facilities, needs and capabilities.
Methodology for verifying the flowed down system requirements and acceptance criteria.
Test tool requirements and development plans.
Risk management plan updated.
Risks that may impact cost, schedule and technical goals completed.
Preliminary configuration management plan is available addressing:
Configuration identification, change control, status accounting, and configuration audits.
Independent Verification and Validation (IV&V) plan available and IV&V assessment of software requirements, if reviewed
SwRR Entrance Criteria - Requirements
Preliminary allocation of system requirements to software available
Preliminary software requirements (SRS)
Complete for preliminary concept.
Requirements are consistent, feasible, testable, and traceable.
Test, delivery, and quality requirements identified and understandable.
Functional requirements.
High-level requirements defined for each functional area.
Block diagram exists for the major software components in each functional area, their interfaces and data flows
Relevant software operational modes defined (e.g., nominal, critical, contingency).
Critical and/or controversial requirements identified, including safety-critical requirements, open issues, and areas of concern.
Requirements identified that need clarification or additional information.
Performance requirements.
Performance requirements for the software identified.
Description exists of critical timing relationships and constraints.
Software requirements and interface requirements have been analyzed and specified.
Quality assurance assessment of the requirements completed and ready for review.
Bidirectional traceability matrix.
Requirements traced to higher-level requirements.
Includes identification of verification methodology (e.g., test, demonstration, analysis, inspection).
Report of current computer resource estimates and margins (memory, bus, throughput) available for review.
Design constraints documented.
Design drivers exist:
Explanation of design drivers and preliminary investigations made during the requirements process to determine reasonableness of the requirements, including preliminary decisions regarding software architecture, operating systems, reuse of existing software, and selection of commercial-off-the-shelf (COTS) components.
Resource goals and preliminary sizing estimates (including timing and database storage) in the context of available hardware allocations; strategies for measuring and tracking resource utilization.
Initial Build Plan.
Review completed for technical and economic feasibility of allocation of functions at the (sub)system level to hardware, firmware, and software.
Results of technical and economic feasibility review and associated analyses.
Bidirectional traceability matrix.
Independent verification and validation (IV&V) plan and assessment of software requirements.
QA assessment of requirements.
Computer resource estimates and margins.
Configuration management (CM) plan.
Software Development Plan (SDP)/Software Management Plan (SMP).
Software concept of operations.
Peer review results.
SwRR Exit/Success Criteria
Review panel agrees that plans and requirements are satisfactory and ready to proceed to the design phase:
Software requirements determined to be clear, complete, consistent, feasible, traceable, testable.
Software Management Plan (SMP), software requirements, interface requirements, verification and validation (V&V) planning is adequate and feasible basis for architectural design activities and are approved, baselined, and placed under configuration management (CM).
Requirements and performance requirements defined, testable, and consistent with cost, schedule, risk, technology readiness, and other constraints.
System requirements, approved material solution, available product/process technology, and program resources form a satisfactory basis for proceeding into the development phase.
Milestones are verifiable and achievable.
Initial computer resource estimate are within margin limits; if not, plans for control of resource margins is deemed adequate to meet margins by preliminary design reveiw (PDR).
All Software Requirements Review (SwRR) Review Item Discrepancies (RIDs) and actions are documented with resolution plans and authorization received to proceed to software architecture design.
Div
id
tabs-5
Mission Definition Review (MDR)
Info
The MDR (or SDR) examines the proposed requirements, the mission/system architecture, and the flow down to all functional elements of the system. (NPR 7120.5
Successful completion of the previous review (typically SRR) and responses made to all Requests for Actions (RFAs) and Review Item Discrepancies (RIDs)
Final agenda, success criteria, and charge to the board have been agreed to by the technical team, project manager, and review chair
Technical products for this review made available to participants prior to MDR
System requirements document updated, if applicable
Concept of operations updated, if applicable
Mission requirements, goals, and objectives updated, if applicable
Preliminary Software Development/Management Plan (SDP/SMP)
Updated risk management plan, if applicable
Updated risk assessment and mitigations (including Probabilistic Risk Assessment (PRA), as applicable)
Updated project software classification(s)
Cost and schedule data updated
Preferred system solution definition exists, including major trades and options
Overall concept is reasonable, feasible, complete, responsive to the mission requirements, and is consistent with system requirements and available resources (cost, schedule, mass, and power)
Software design approaches and operational concepts exist and are consistent with the requirements set
Requirements, design approaches, and conceptual design will fulfill the mission needs within the estimated costs
Major risks have been identified and technically assessed, and viable mitigation strategies have been defined
System level requirements are clearly and logically allocated to software
Div
id
tabs-6
System Definition Review (SDR)
Info
The MDR (or SDR) examines the proposed requirements, the mission/system architecture, and the flow down to all functional elements of the system. (NPR 7120.5
Successful completion of the previous review (typically SRR) and responses made to all Requests for Actions (RFAs) and Review Item Discrepancies (RIDs)
Final agenda, success criteria, and charge to the board have been agreed to by the technical team, project manager, and review chair
Technical products for this review made available to participants prior to SDR
Preferred software solution defined including major tradeoffs and options
Baselined documentation updated, as required
Preliminary functional baseline (with supporting trade-off analyses and data) available
Preliminary system software functional requirements available
As applicable, risk management plan updated (could be part of SDP/SMP)
Updated software risk assessment and mitigations (including Probabilistic Risk Assessment (PRA), as applicable)
As applicable, SDP/SMP updated
Updated technology development, maturity, and assessment plan
Updated cost and schedule data
Work Breakdown Structure
Preliminary software safety analysis available for review
Project software data dictionary available
Preliminary project software assurance plan available
As applicable, updated project software maintenance plan available
Software inputs / contributions completed for:
Systems Engineering Management Plan (SEMP) changes, if any
Based on system complexity, updated human rating plan
Flow down of system requirements to all software functional elements of the system
Requirements process
Technical approach
SDR Items Reviewed
System architecture, including software
Preferred software solution with tradeoffs and options
Preliminary functional baseline
Preliminary system software functional requirements
Risk management plan, as applicable
SDP/SMP, as applicable
Preliminary software verification and validation (V&V) planning, as applicable
Software requirements documents
Interface requirements documents, including SW
Technical resource utilization estimates and margins
Software safety analysis
Software data dictionary
Software configuration management (CM) plan
Software quality assurance (QA) plan
SDR Exit/Success Criteria
Review panel agrees that:
Software requirements, including mission success criteria and any sponsor-imposed constraints, are defined and form the basis for the proposed conceptual design
System level requirements are flowed down to software
All software technical requirements are allocated and flow down to subsystems is adequate; they are verifiable, and traceable to their corresponding system level requirement; requirements, design approaches, and conceptual design will fulfill the mission needs consistent with the available resources (cost, schedule, throughput, and sizing)
Technical plans have been updated, as necessary, including risk management plan, SDP/SMP, V&V planning, software maintenance plan, QA plan, CM plan
Tradeoffs are completed, and those planned for Phase B adequately address the option space
Adequate planning exists for the development of any enabling new technology
Significant development, mission, and safety risks are identified and technically assessed, and a process and resources exist to manage the risks
Operations concept is consistent with proposed design concept(s) and in alignment with the mission requirements
Requisite level of detail and resources are available to support the acquisition and development plan within existing constraints
All of these software subsystem requirements are traceable to either mission objectives, concept of operations, or interface requirements
Monitoring processes/practices are in place to create software subsystem within planned technical, schedule, cost, effort, and quality capabilities
Preliminary verification approaches are agreed upon
Div
id
tabs-7
Preliminary Design Review (PDR)
Info
The PDR demonstrates that the preliminary design meets all system requirements with acceptable risk and within the cost and schedule constraints and establishes the basis for proceeding with detailed design. It shows that the correct design option has been selected, interfaces have been identified, and verification methods have been described. Full baseline cost and schedules, as well as risk assessments, management systems, and metrics are presented. (NPR 7120.5
Successful completion of the SDR or MDR and responses made to all SDR or MDR Requests for Actions (RFAs) and Review Item Discrepancies (RIDs), or a timely closure plan exists for those remaining open
Final agenda, success criteria, and charge to the board have been agreed to by the technical team, project manager, and review chair
Technical products for this review made available to participants prior to PDR
Baselined documentation updated, as required
Risk assessment and mitigation updated
Cost and schedule data baselined
Peer reviews completed: SRS, software architectural design (if identified for SW peer review/inspection in SW development plans), integration test plans
Lessons Learned
Review of existing Lessons Learned (LL) from previous projects completed
Lessons Learned captured from software areas of the project (indicate the problem or success that generated the LL, what the LL was, and its applicability to future projects)
Confirmation exists that Lessons Learned added to LL database
PDR Entrance Criteria - Plans
Applicable technical plans (e.g., technical performance measurement plan, payload-to-carrier integration plan, producibility / manufacturability program plan, reliability program plan, baselined quality assurance plan) available
SDP/SMP baselined
Work Breakdown Structure
For the corresponding detailed design activities
Metrics established and gathered to measure software development progress
Logistics documentation (e.g., maintenance plan) updated, as required
Software inputs or contributions completed for:
Updated Project Plan
Updated Technology Development Maturity Assessment Plan
Configuration management plan baselined
Configuration Control Board established for software (and change control procedures working)
Configuration management system understood by those who must use it
Procedures and tools developed for mechanizing management and configuration management plans
Supplier documentation available for review:
Software Data Dictionary(s)
Software Classification(s)
SDP/SMP [with verification and validation (V&V) separate]
Software configuration management plan(s)
Software assurance plan(s)
Software maintenance plan(s)
Test tools and facility requirements identified with plans and actions to ensure their availability when needed
Architectural design (baselined) verified via operational scenarios to include required functionality, operating modes, and states
Results available from evaluations of prototype software, if necessary to evaluate design
Human engineering aspects of design addressed with solutions acceptable to potential users
SDD and traceability matrix review by test team completed and SDD updated as needed
Critical components identified and trial coding scheduled
Confirmation exists that
The test group participated in requirements and design analysis
Interdisciplinary teams are working design issues that cross (sub)system component boundaries (software, hardware, etc.)
PDR Entrance Criteria - Analysis
Safety analyses and plans baselined:
Matrix showing each subsystem/task/component's software classification (per NPR 7150.2), its safety classification (per NASA-STD-8719.13B), the rationale for the classifications, and the status of the classifications' approval by Software Assurance and management
Updated PHA, Software Safety Litmus Test, if necessary
Documented solutions, analysis, decisions and rationale
Completed analyses
Prototype software, if applicable
Plans for development and test tools and facilities
Software development progress metrics
CM plan
Peer review results / proof of completion
Status of change requests
PDR Exit/Success Criteria
Top-level requirements including mission success criteria, Technical Performance Measures (TPMs), and any sponsor-imposed constraints are agreed upon, finalized, stated clearly, and consistent with preliminary design
Review panel agrees that:
Flow down of verifiable requirements is complete and proper or, if not, an adequate plan exists for timely resolution of open items; requirements are traceable to mission goals and objectives
All supplier software requirements are verifiable
Preliminary design is expected to meet the functional and performance requirements at an acceptable level of risk
Definition of technical interfaces is consistent with overall technical maturity and provides an acceptable level of risk
Adequate technical interfaces are consistent with the overall technical maturity and provide an acceptable level of risk
Adequate technical margins exist with respect to TPMs
Any required new technology has been developed to an adequate state of readiness, or back-up options exist and are supported to make them a viable alternative
Project risks are understood and credibly assessed; plans, process, and resources exist to effectively manage them
Operational concept is technically sound, includes (where appropriate) human factors, and includes flow down of requirements for its execution
Proposed design approach has sufficient maturity to proceed to final design
Subsystem requirements, subsystem preliminary design, results of peer reviews, and plans for development, testing and evaluation form a satisfactory basis for proceeding into detailed design and test procedure development
SMP, the software architectural design, and integration test plans adequate and feasible to support software detailed design
All RIDs/actions are completed or have closure plans and customer approval received to proceed to detailed design phase
Products from this review are approved, baselined and placed under configuration management
Approval received for software inputs / contributions:
Safety and mission assurance (e.g., safety, reliability, maintainability, quality, and IEEE parts) is adequately addressed in preliminary designs and any applicable S&MA products (e.g., Probabilistic Risk Assessment (PRA), system safety analysis, and failure modes and effects analysis)
Management processes used by the mission team are sufficient to develop and operate the mission
Cost estimates and schedules indicate that the mission will be ready to launch and operate on time and within budget, and that the control processes are adequate to ensure remaining within allocated resources
Div
id
tabs-8
Critical Design Review (CDR)
Info
The CDR demonstrates that the maturity of the design is appropriate to support proceeding with full scale fabrication, assembly, integration, and test, and that the technical effort is on track to complete the flight and ground system development and mission operations in order to meet mission performance requirements within the identified cost and schedule constraints. Progress against management plans, budget, and schedule, as well as risks assessments are presented. (NPR 7120.5
Successful completion of the previous review (typically PDR) and responses made to all Requests for Actions (RFAs) and Review Item Discrepancies (RIDs), or a timely closure plan exists for those remaining open
Final agenda, success criteria, and charge to the board have been agreed to by technical team, project manager, and review chair
Technical products for this review made available to participants prior to CDR
Baselined documents updated, as required
Peer reviews for software and rework accomplished, as defined in the s/w and/or project plans
NPR 7150.2 compliance matrix baselined
Lessons Learned
Review of existing Lessons Learned (LL) from previous projects completed
Lessons Learned captured from software areas of the project (indicate the problem or success that generated the LL, what the LL was, and its applicability to future projects)
Confirmation exists that Lessons Learned added to LL database
CDR Entrance Criteria - Plans
Software development/management plan (SDP/SMP) updated for implementation and unit test activities
Updated Work Breakdown Structure
Updated cost and schedule data
Progress against software management plans available for review
Plan exists for milestone and peer reviews, walkthroughs, and external reviews
Documentation plan exists, including each document's status and when it will be baselined
Software requirements, management process, including documents used and produced, and verification and validation (V&V) planning updated and baselined
Staffing-up problems being addressed, contingency plans in place
Independent verification and validation (IV&V) plans and status available for review
Management procedures and tools for measuring and reporting progress available and working
Software measurements available for review (on planned and actual regarding product size, cost, schedule, effort, and defect)
Procedures established and working for software quality assurance and quality an integral part of the product being produced
Implementation process exists, incl. standards, review process, problem reporting, unit test, integration
Supplier documentation available for review:
Software Design Description(s)
Interface Design Description(s)
Updated Supplier Software V&V Plan(s)
Preliminary Supplier Software Test Procedure(s)
Systems and subsystem certification plans and requirements exist (as needed)
Changes since PDR available for review:
Updated product assurance and software safety plans and activities
System safety analysis with associated verifications
Status of configuration management processes since PDR available, including discrepancy reporting and tracking (development and post-release)
Build plan exists
Build test timeline and ordered list of components and requirements to be tested in each build ready for review
Test group trained and prepared to evaluate the code using their facilities and tools
Software is ready for testing / activation
Coding, integration, and test plans and procedures available for review
Test plans baselined
Preliminary integration and test procedures
Test team roles, functions, support required are defined
Test levels described (e.g., unit testing, integration testing, software system testing) – description, who executes, test environment, standards followed, verification methodologies
Testing preparation and execution activities planned, incl. testing of reused/heritage software, if applicable
Test environments described for each test level --diagram and description of tools, testbeds, facilities
Plans available for review:
Launch site operations plan
Checkout and activation plan
Disposal plan (including decommissioning or termination)
System and acceptance testing defined – operational scenarios to be tested, including stress tests and recovery testing, if applicable
Acceptance process exists – reviews (e.g., Acceptance Test Readiness Review, Acceptance Test Review), approval, and signoff processes
Acceptance criteria baselined
Software inputs / contributions completed for:
Updated Project Plan
Updated technology development maturity assessment plan
CDR Entrance Criteria - Requirements
Changes to IT security requirements since PDR available for review (Mission-specific)
SRS updated to the Computer Software Unit (CSU) level
Traceability matrix updated (to CSU level)
Verification exists that detailed designs cover the requirements
CDR Entrance Criteria - Design
Technical data package (e.g., integrated schematics, spares provisioning list, interface control documents, engineering analyses, and specifications) available for review
Design process exists, including methodology and standards used, design documentation produced, inspections and reviews
Software design document(s) baselined (including interface design documents, detailed design and unit test)
Command and telemetry list available for review
Final design solution, evaluation, and rationale available
Documented make, buy, and/or reuse, analysis, criteria, and rationale
Reused/heritage software or functionality from previous projects; necessary modifications
Final architecture definition available
Subsystem/component context diagram available
Data flow diagrams available
Software subsystem design diagram available (e.g., Level 0 data flow diagram or Unified Modeling Language (UML))
For each task in the software subsystem design diagram
Design diagrams for the task
Description of functionality and operational modes
Safety considerations addressed in the design
Resource and utilization constraints (e.g., CPU, memory); how the software will adapt to changing margin constraints; performance estimates
Data storage concepts and structures
Input and output data and formats identified
Interrupts and/or exception handling available, including event, FDC, and error messages
IT Security features (design features) identified
Detailed description of software operation and flow exists
Operational limits and constraints identified
Technical resource utilization estimates and margins updated
Detailed timing and storage allocation compiled
Algorithms exist sufficient to satisfy their requirements
Failure detection and correction (FDC) requirements, approach, and detailed design available for review
Trial code analyzed and designs modified accordingly
Designs comprising the software completed, peer reviewed and placed under change control
CDR Entrance Criteria - Analysis
Analyses completed:
Algorithm accuracy
Critical timing and sequence control
Undesired event handling
Operability
Failure modes and effects analyses
Final status and results of analyses ready for review
Subsystem-level and preliminary operations safety analyses exist
Risk assessment and mitigation updated
Reliability analyses and assessments updated
Operational Concepts updated
Product build-to specifications exist for each hardware and software configuration item, along with supporting trade-off analyses and data
Status of change requests available for review
CDR Entrance Criteria - Other
Software requirement verification recording, monitoring, and current status available for review – databases and test reports; sample test verification matrix
Preliminary operations handbook created
Programmer's manual drafted
User's manual drafted
CDR Items Reviewed
Baselined documents
Technical data package
SDP/SMP
Progress against software management plans
Plan and status for reviews
Documentation plan
NPR 7150.2 compliance matrix
Design and implementation processes
Status of management procedures and tools
Software measurements
Logistics documentation (e.g., maintenance plan)
Status of any staffing problems
Software design document(s)
Command and telemetry list
Final Design Solution, Evaluation, and Rationale
Final Architecture Definition
Software subsystem design diagram
Data flow diagrams
Identification and formats of input and output data
Interrupts and/or exception handling, including event, FDC, and error messages
IT Security requirements and features
Detailed description of software operation and flow
Operational limits and constraints
Technical resource utilization estimates and margins
Status and results of analyses
Algorithms sufficient to satisfy their requirements
Failure detection and correction (FDC) requirements, approach, and detailed design
Subsystem/component context diagram
Status of trial code analysis and design
Supplier documentation
Status of SW designs and requirements coverage verification
SRS
Bidirectional Traceability Matrix
Status of software QA and safety plans, procedures, activities
Hazard analysis / Software Assurance Classification Report (SACR), if necessary
Risk assessment and mitigation
Reliability analyses and assessments
IV&V plans and status
Systems and subsystem certification plans and requirements (as needed)
CM processes
Status of development environment and personnel training
Build plan
Product build-to specifications along with supporting trade-off analyses and data
Coding, integration, and test plans and procedures
V&V planning
Software test plan, procedures
Testing preparation and execution activities
Build test timeline and ordered list of components and requirements to be tested in each build
Description of test environments for each test
Status of test group training
Launch site operations plan
Checkout and activation plan
Disposal plan
Preliminary Operations Handbook
Draft of Programmer's Manual
Draft of User's Manual
Status of change requests
CDR Exit/Success Criteria
Review panel agrees that:
All supplier software requirements have been mapped to the software design
All elements of the design are compliant with functional and performance requirements (detailed design is expected to meet requirements with adequate margins at acceptable level of risk)
Interface control documents are sufficiently matured to proceed with fabrication, assembly, integration, and test, and plans are in place to manage any open items
Product verification and product validation requirements and plans are complete; verification approach is viable, and will confirm compliance with all requirements
Management processes used by the project team are sufficient to develop and operate the mission
Testing approach is comprehensive, and planning for system assembly, integration, test, and launch site and mission operations is sufficient to progress into next phase
Adequate technical and programmatic margins and resources exist to complete development within budget, schedule, and risk constraints
Risks to mission success are understood and credibly assessed, and plans and resources exist to effectively manage them
SDP/SMP, software detailed designs, and unit test plans are an adequate and feasible basis for the implementation and test activities
High confidence exists in the product baseline, and adequate documentation exists or will exist in a timely manner to allow proceeding with coding, integration, and test
Approval received for software inputs / contributions:
Safety and mission assurance (e.g., safety, reliability, maintainability, quality, and EEE parts) have been adequately addressed in system and operational designs, and any applicable S&MA products (e.g., Probabilistic Risk Assessment (PRA), system safety analysis and failure modes and effects analysis) have been approved
High priority RIDs against the SDD are closed/actions are completed and customer approval received to proceed to next phase
Approved readiness to proceed with software implementation and test activities
Products from this review are approved, baselined, and placed under configuration management
Div
id
tabs-9
Production Readiness Review (PRR)
Info
The PRR is held for projects developing or acquiring multiple similar or identical flight and/or ground support systems. The purpose of the PRR is to determine the readiness of the system developer(s) to efficiently produce (build, integrate, test, and launch) the required number of systems. The PRR also evaluates how well the production plans address the system's operational support requirements. (NPR 7120.5
Significant production engineering problems encountered during development are resolved
Design documentation is adequate to support production
Production plans and preparation are adequate to begin fabrication
Production-enabling products and adequate resources are available, allocated, and ready to support end product production
Production risks and mitigations identified
Schedule reflects production activities
PRR Items Reviewed
Design documentation
Production plans and preparation
Production risks and mitigations
Schedule
PRR Exit/Success Criteria
Review panel agrees that:
System requirements are fully met in final production configuration
Adequate measures are in place to support production
Design-for-manufacturing considerations ensure ease and efficiency of production and assembly
Risks are identified, credibly assessed, and characterized, and mitigation efforts defined
Alternate sources for resources identified, as appropriate
Required facilities and tools are sufficient for end product production
Specified special tools and test equipment are available in proper quantities
Production and support staff are qualified
Production engineering and planning are sufficiently mature for cost-effective production
Production processes and methods are consistent with quality requirements
Qualified suppliers are available for materials that are to be procured
Delivery schedules are verified
Div
id
tabs-10
System Integration Review (SIR)
Info
The SIR evaluates the readiness of the project to start flight system assembly, test, and launch operations. V&V Planning, integration plans, and test plans are reviewed. Test articles (hardware/software), test facilities, support personnel, and test procedures are ready for testing and data acquisition, reduction, and control. (NPR 7120.5
Integration plans and procedures completed and approved
Segments and/or components available for integration
Mechanical and electrical interfaces verified against the interface control documentation
All applicable functional, unit-level, subsystem, and qualification testing conducted successfully
Integration facilities, including clean rooms, ground support equipment, electrical test equipment, and simulators ready and available
Support personnel adequately trained
Handling and safety requirements documented
All known system discrepancies identified and disposed in accordance with agreed-upon plan
All previous design review success criteria and key issues satisfied in accordance with agreed-upon plan
Quality control organization is ready to support integration effort
SIR Items Reviewed
Integration plans and procedures
Interface control documentation
Functional, unit-level, subsystem, and qualification test results/proof of completion
Test preparation (facilities, tools, equipment, personnel)
Handling and safety requirements
V&V planning, test plans
SIR Exit/Success Criteria
Review panel agrees that:
Adequate integration plans and procedures are completed and approved for the system to be integrated
Previous component, subsystem, and system test results form a satisfactory basis for proceeding to integration
Integration procedures and work flow have been clearly defined and documented
Review of integration plans, as well as procedures, environment, and configuration of items to be integrated, provides a reasonable expectation that integration will proceed successfully
Integration personnel have received appropriate training in integration and safety procedures
Risk level is identified and accepted by program/project leadership, as required
Div
id
tabs-11
Test Readiness Review (TRR)
Info
The TRR ensures that the test article (hardware/software), test facility, support personnel, and test procedures are ready for testing and data acquisition, reduction, and control. (NPR 7123.1
Required documents are in the state/status required; any required deviations or waivers are in place and approved
All known system discrepancies identified and disposed in accordance with agreed-upon plan
Software cost estimate updated, and software related expenditures collection and report by life cycle phases available
Test schedule updated:
Current system test status
Issues and concerns
Test schedule
Schedules for integration and test established and are reasonable based on results of unit testing
Lessons Learned
Plans to capture any lessons learned from test program are documented
Review of existing Lessons Learned from previous projects completed
Lessons Learned captured from software areas of the project ( indicate the problem or success that generated the Lesson Learned, what the Lesson Learned was, and its applicability to future projects)
TRR Entrance Criteria - Plans
Objectives of testing and testing approach clearly defined and documented and supported by:
Test plans, test cases, procedures, environment
Defined configuration of test item(s)
Interfaces placed under configuration management or defined in accordance with an agreed-to plan
All required test resources, people (including a designated test director), facilities, test articles, test instrumentation, and other test enabling products identified and available to support required tests
Facilities and tools for integration and test ready, qualified, validated, and available for operational use including test engineering products (test cases, procedures, tools, etc.), test beds, simulators, and models
Roles and responsibilities of all test participants defined and agreed to and all personnel have been trained
Supplier Software Version Description(s) available
Metric data and reports (implementation and test) ready for review
SDP/SMP updated for integration and test activities
IV&V report/status - if applicable
Any current risks, issues, or requests for action (RFAs) that require follow-up and how they will be tracked to closure ready for review
Risk analysis and risk list updated and associated risk management plan updated
Test plan includes test scenarios:
User-defined scenarios to test interactive or operator-oriented software
Safety critical scenarios
For all software/system requirements defined in the bidirectional traceability matrix
Performance checks at the limit of ranges specified for the requirements and operational scenarios, including test limitations and constraints
Test case structure established that identifies per test case:
Software requirements to be tested and SW entities to be exercised
Required inputs
Facilities and test tools required, setup, and required qualifications
Limitations of the test environment
Verification of standards compliance
Test plan updated for integration and test activities
Software test procedures baselined:
Defined build process
Safety critical software test considerations
Defined software test success criteria
Defined CM process and procedures used for testing
Process for capturing test data and storing it
Test procedure red-line process
Process for restarting a test if error found during testing
Discrepancy Reporting System
Process for tracking Test Progress
Role of Quality Assurance including redlining and QA witnessing role and responsibilities
Any safety and security issues relevant to the testing activity
All workarounds and non-functioning software components
Time required for testing; include schedule and analysis of time needed on various environments / test beds / spacecraft
TRR Entrance Criteria - Requirements
All requirements included in baselined test procedure document and uniquely identified and traceable in the updated bidirectional traceability matrix (includes necessary corrections due to discrepancy reports)
TRR Entrance Criteria - Design
All previous design review success criteria and key issues satisfied in accordance with agreed-upon plan
TRR Entrance Criteria - Analysis
Outstanding software change requests (SCRs) ready for review
Code inspection results available for review
Results of testing completed to date available for review:
Objectives of tests
Expected results defined
Confirm all steps in the test runs are documented
Results including safety critical test results
Tests performed
Successful tests
Known problems, issues
Deviations, waivers
TRR Entrance Criteria - Other
Software build created from CM and ready for testing
Applicable functional, unit-level, subsystem, system, and qualification testing conducted successfully
Informal dry run completed without errors
Validation of operations and users manuals completed
Successful functional audit (FCA) of the version description document (VDD) (such as FSW) including fixes
Tests reusable for regression testing exist
Databases for integration and test have been created and validated
Test network showing interdependencies among test events and planned time deviations for these activities prepared
Evaluations completed (in conjunction with unit testing):
Verification of computations using nominal and stress data
Evaluations completed (in conjunction with s/w integration and test):
Verification of performance throughout anticipated range of operation conditions including nominal, abnormal, failure and degraded mode situations
Verification of performance throughout anticipated range of operating conditions as various strings of units are linked together and various modes are exercised
Verification of end-to-end functional flows and database linkages
Exercise of logic switching and executive control options at least once
TRR Items Reviewed
Test preparation
Test plans, test cases, scenarios, databases, procedures, environment, expected results, and configuration of test item(s)
Software build ready for testing
Resources (people, facilities, tools, etc.)
Test schedule
Test contingency planning
Test network
Results for all testing completed to date
Interfaces
SDP/SMP
VDD and VDD audit results
Software change requests
Bidirectional traceability matrix
Current risks, issues, or requests for action (RFAs)
Baselined documentation from previous reviews
Requirements and design
Status of quality assurance (QA) activities
Status of known system discrepancies
Software cost estimate and expenditures report
Supplier Software VDD(s)
Requirements Analysis and Traceability Reports
Code Analysis and Assessment Results
Metric Data and Reports
Operations and users manuals
Completed evaluations of unit, integration tests
Risk analysis, list, management plan
TRR Exit/Success Criteria
Review panel agrees that:
Peer reviews completed for implementation and tests to be performed, as defined in the software plans
Adequate identification and coordination of required test resources are completed
Previous component, subsystem, and system test results form a satisfactory basis for proceeding into planned tests
All the entrance criteria have been met
Test cases have been reviewed and analyzed for expected results, and results are consistent with test plans and objectives
Test personnel have received appropriate training in test operation and safety procedures
Provisions have been made should test levels or system response exceed established limits or if the system exceeds its expected range of response
Software is ready to be tested
SDP/SMP, software implementations, and test plans are an adequate and feasible basis for integration and test activities
Formal dry test run completed
Adequate test plans are completed and approved to proceed for the system under test
Risk level associated with testing is identified (during TRR) and accepted by the appropriate program/competency leadership, as required
Products from this review are approved, baselined and placed under configuration management
Div
id
tabs-12
System Acceptance Review (SAR)
Info
The SAR verifies the completeness of the specific end item with respect to the expected maturity level and to assess compliance to stakeholder expectations. The SAR examines the system, its end items and documentation, and test data and analyses that support verification. It also ensures that the system has sufficient technical maturity to authorize its shipment to the designated operational facility or launch site. (NPR 7120.5
Technical products for this review made available to participants prior to SAR
Acceptance test readiness available for review
Process for analysis of Test Results including the division of responsibility
Acceptance Test testbed (environment) setup (hardware)
Setup and use of Simulators or other Test tools and their required qualifications
Limitations of the testbed (environment)
Tests that require: hardware for verification and/or human input
Description, at a high level, of what each test does, how long it lasts, and any special circumstances
IV&V report/status - if applicable
Preparedness for Acceptance Testing
Requests For Action (RFAs)
Decision to proceed to Acceptance Testing
Results available from SARs conducted at major suppliers
Transition to production and/or manufacturing plan exists
Product verification results / final test reports available
Product validation results available
Acceptance plans and acceptance criteria
Documentation exists to confirm that the delivered system complies with the established acceptance criteria
Documentation exists to confirm that the system will perform properly in the expected operational environment
Technical data package updated to include all test results
Certification package available for review
Risk assessment and mitigations updated
Previous milestone reviews successfully completed
Metrics data and reports available for review
Remaining liens or unclosed actions and plans for closure available for review
Waivers and deviations available for review
Software build has been updated
Functional audit (FCA) completed
Software presentation prepared (for AR):
Software overview
Project System Diagram
Functional software overview
Software products/artifacts
Software traceability matrix examples
Software Test Procedures status
Open RIDs
Open SCRs
Software summary and recommendations
Lessons Learned
Review of existing Lessons Learned from previous projects completed
Lessons Learned captured from software areas of the project ( indicating the problem or success that generated the Lesson Learned, what the Lesson Learned was, and its applicability to future projects)
Confirmation exists that Lessons Learned added to Lessons Learned database
SAR Items Reviewed
Test readiness information
Results of the SARs conducted at the major suppliers
Transition to production and/or manufacturing plan
Product verification results / test reports
Product validation results
Baselined Software Build
Certification package
Documentation that the delivered system complies with the established acceptance criteria
Documentation that the system will perform properly in the expected operational environment
Technical data package
Risk assessment and mitigation
Hazard report
Results/proof of completion for previous milestone reviews
Remaining liens or unclosed actions and plans for closure
Waivers/deviations
Metrics Data and Reports
SAR Exit/Success Criteria
Review panel agrees that:
Required tests and analyses are complete and indicate that system will perform properly in expected operational environment
Risks are known and manageable
Software system meets established acceptance criteria
Required safe shipping, handling, checkout procedures are complete and ready for use
Required operational plans and procedures are complete and ready for use
Technical data package is complete and reflects delivered system, including software user's manual and version description document
All applicable lessons learned for organizational improvement and system operations are captured
Software system has sufficient technical maturity to authorize shipment to designated operational facility or launch site
Div
id
tabs-13
Operational Readiness Review (ORR)
Info
The ORR examines the actual system characteristics and the procedures used in the system or product's operation and ensures that all system and support (flight and ground) hardware, software, personnel, and procedures are ready for operations and that user documentation accurately reflects the deployed state of the system. (NPR 7120.5
Test failures and anomalies from validation testing resolved and results incorporated into all supporting and enabling operational products
All operational supporting and enabling products (e.g., facilities, equipment, documents, updated databases) that are necessary for the nominal and contingency operations have been tested and delivered/installed at the site(s) necessary to support operations
Operations manual complete
Physical audit (PCA) completed
Software inputs / contributions completed for:
Training provided to users and operators on correct operational procedures for system
Ground systems readiness
Diagram describing main functionality for project, how parts interact, and main flow of data between major functional parts
Problem reporting and change request process for discrepancy reports (DR), enhancement reports (ER), Database change requests (DCR)
Current DR, ER, DCR status, include historical trend data, and details on current open DRs, ERs, DCRs
Key parts of system, their current operational readiness, and how verified
Any issues, how they will be handled, and workarounds available including when permanent fixes will be completed
Key interactions with other systems, their operational readiness, and how verified; any issues, how they will be handled, and workarounds available including when permanent fixes will be completed
Outstanding items that need to be completed before readiness is achieved along with scheduled date
Software maintenance plan completed
When software is frozen, what types of fixes will be approved for implementation under a freeze, etc.
How change control board (CCB) will handle software changes or bug fixes
Science planning and processing system readiness is available for review, as applicable:
Diagram describing science data processing products and general timelines involved
Diagram describing science system context (relationship of main Mission Operations Center, Mission Planning Office, Science Validation Facility, Ground stations, interconnecting networks, and the main science data Instrument teams)
Description of these main components in high-level detail including planning and processing functions; include any special cases for launch, in-orbit checkout, end of mission, etc.; description of testing, results, and issues done to verify and validate these components
Summary of all testing done, results, and outstanding issues for Science Data Processing
Safety and security issues status available for review:
Software issues with safety, how addressed, and current status
Software issues with security, how addressed, and current status
Simulations status available for review:
Number and main details for simulations by subsystem exercised, for example: Launch, Attitude Control System, Command & Data Handling, Communication, Flight Software, Power System Electronics, Mission Operations Center, Pre-Launch, others deemed important for project
Outstanding issues from Simulation testing, schedule impact, workarounds, and risks; for workarounds, when will problem/issue be permanently fixed
Contingencies and constraints available for review:
State of Contingency Flow Chart Book and any planned updates
List of current constraints on system, state of database that details these constraints, and any outstanding actions that need to be taken
Audits that were done and against what areas to verify constraints
Operational problem escalation process
Operational emergency notification process including telephone numbers to be called
Status of documentation readiness available for review:
Version Description Document(s); its location, and any outstanding issues
Baselined Software User's Manual; its location, and any outstanding issues
Software Operations Plan; its location, and any outstanding issues
Software Maintenance Plan; its location, and any outstanding issues
Planned software retirement activities; location, and any outstanding issues
Lessons Learned
Lessons Learned (LL) captured from software areas of the project (indicating the problem or success that generated the Lesson Learned, what the LL was, and its applicability to future projects)
Confirmation exists that Lessons Learned added to LL database
Status of work remaining available for review:
All critical work that needs to be completed along with expected completion data
ORR Items Reviewed
Validation test results/proof of completion
Status of test failures and anomalies from validation testing
Status of all testing, delivery, and installation for operational supporting and enabling products necessary for nominal and contingency operations
Status of software user's manual
Status of operations manual
Software Maintenance Plan
Science Planning and Processing System Readiness
Safety and Security Issues
Number and main details for simulations by subsystem exercised and any open issues
Contingencies and constraints
Status of documentation readiness
Work Remaining
ORR Exit/Success Criteria
Review panel agrees that:
System, including any enabling products, is ready to be placed in operational status
All applicable lessons learned for organizational improvement and systems operations have been captured
All waivers/deviations and anomalies have been closed
Systems hardware, software, personnel, and procedures are in place to support operations
All project and support h/w, s/w, and procedures are ready for operations and user documentation accurately reflects the deployed state of the entire system
RFA and review item discrepancy (RID) reports generated, as needed, as result of this ORR
Div
id
tabs-14
Flight Readiness Review (FRR)
Info
The FRR examines tests, demonstrations, analyses, and audits that determine the system's readiness for a safe and successful flight/launch and for subsequent flight operations. It also ensures that all flight and ground hardware, software, personnel, and procedures are operationally ready. (NPR 7120.5
Certification received that flight operations can safely proceed with acceptable risk
System and support elements confirmed as properly configured and ready for flight
Interfaces compatible and function as expected
System state supports a launch "go" decision based on go/no-go criteria
Flight failures and anomalies from previously completed flights and reviews resolved and results incorporated into all supporting and enabling operational products.
System configured for flight
Tests, demonstrations, analyses, audits support flight readiness
FRR Items Reviewed
Open items and waivers/deviations
System and support elements configuration confirmation
Status of interface compatibility and functionality
System state
Status of failures and anomalies from previously completed flights and reviews
System configuration
Tests, demonstrations, analyses, audits
Software user's manual
FRR Exit/Success Criteria
Review panel agrees that:
Flight vehicle is ready for flight
Software is deemed acceptably safe for flight (i.e., meeting the established acceptable risk criteria or documented as being accepted by the PM and Designated Governing Authority (DGA))
Flight and ground software elements are ready to support flight and flight operations
Interfaces are checked and found to be functional
Open items and waivers/deviations have been examined and found to be acceptable
Software contributions to all open safety and mission risk items have been addressed
Operators are ready and work-arounds have been fully vetted
Software user's manual is ready and available to be used for testing
