1. Requirements

3.1.5 The project manager shall define and document the acceptance criteria for the software. 

1.1 Notes

NPR 7150.2, NASA Software Engineering Requirements, does not include any notes for this requirement.

1.2 History

1.3 Applicability Across Classes

Key:    - Applicable | - Not Applicable

2. Rationale

Acceptance criteria synchronize the visions of the client and the development team. They ensure that everyone has a common understanding of the requirements: Developers know exactly what kind of behavior the feature must demonstrate, while stakeholders and the client understand what's expected from the feature.

3. Guidance

"Acceptance criteria" are defined as (1) those criteria that a system or component must satisfy to be accepted by a user, customer, or other authorized entity (ISO/IEC/IEEE 24765:2010 Systems and software engineering – Vocabulary ²³⁰), and, (2) those criteria, including performance requirements and essential conditions, which must be met before project deliverables are accepted (A Guide to the Project Management Body of Knowledge ³⁰²). The purpose of this requirement is to cause the software development team to work with the customer up front to define the criteria to be used for software acceptance. The acceptance criteria and timing can help to do the following:

• Define the software support manpower levels on a project.
• Determine the "handover" from the software development organization to a software operation and maintenance organization.
• Help define what a contractor is to complete before flight certification can be achieved.

The software acceptance criteria definition is a key element in the overall software planning process. The software acceptance criteria need to address both software and data. If the software work product is delivered in phases, each delivery may have its acceptance criteria. Software development teams can better organize their development tasks if they have well-defined plans for software activities. Customers and project personnel must include pre-determined criteria in these plans to evaluate the software work products. These criteria include clearly understanding what the customer wants the product to do and what constitutes an acceptable software product. The development team documents the agreed-to acceptance criteria and the planned activities in a software acceptance plan contained in the Software Development/Management Plan (see 5.08 - SDP-SMP - Software Development - Management Plan) or a separate Verification and Validation (V&V) Plan, as a formal means of guiding these activities.

Acceptance activities for software development begin with the planning and the development of acceptance criteria during the Formulation phase of the project. These activities and acceptance criteria can be documented in the Software Development/Management Plan (see 5.08 - SDP-SMP - Software Development - Management Plan ) or a separate Software V&V Plan. As an understanding of the system grows, and the requirements are better understood, the acceptance criteria can be reviewed to assure they are up to date and consistent with the system being built. They typically conclude with the system acceptance review late in the Implementation phase of the project (see the entrance and exit criteria for the Systems Acceptance Review in 7.09 - Entrance and Exit Criteria )

For software acquired via the acquisition process (see 7.03 - Acquisition Guidance ) the acceptance criteria development follows this guidance. The final criteria must be put into the contract statement of work.

The paragraphs below deal with the acceptance criteria creation and documentation. Acceptance testing, which generates the information used to assess the satisfaction of the acceptance criteria, is also discussed.

Acceptance Criteria are used in the delivery process, see SWE-077 - Deliver Software Products, SWE-191 - Software Regression Testing, 

3.1 Establish Acceptance Criteria

The software lead engineer works with the customer and the software development team to develop the appropriate acceptance plans and activities to ensure that functional requirements are properly translated into system acceptance criteria. A well-defined acceptance plan will help the software development team and the software assurance team identify the resources required to properly support the effort. This, in turn, contributes to efficient and cost-effective development and V&V activities, where many of the acceptance activities occur. The team defines acceptance criteria during the Formulation phase of the software development life cycle to assure sufficient time to prepare for and perform the acceptance activities.

The software development team and the customer work together to ensure that they do the following:

• Identify interim and final products that will be part of acceptance activities.
• Develop the acceptance criteria and activities schedule.
• Plan how and by whom each acceptance activity will be performed.
• Schedule adequate time for the customer to examine and review the product.
• Prepare the acceptance plan.
• Perform formal acceptance testing at scheduled times.
• Plan the decision-making process that is based on the results of acceptance testing.

The software lead engineer works with the software team (including the software assurance personnel) to develop appropriate product reviews, tests, and any audits necessary. This work includes identifying:

• The types of criteria to consider, such as customer expectations and requirements, technology limitations, environmental impact, safety, risks, total ownership, life cycle costs, and schedule impact.
• The variations in criteria for computer software configuration items (CSCI), units, and or systems.
• The acceptable range of the criteria.
• The rank of each criterion by its importance.

Acceptance criteria may include:

• Product V&V was completed successfully.
• V&V of individual products, integration of products into systems, and that system V&V has been performed or witnessed by the technical team.
• The technical data package is current (as-built) and complete.
• Transfer of certifications, warranties, or representations is complete.
• Transfer of software products, licenses, data rights, intellectual property rights, etc., is complete.
• If an acquisition: Technical documentation required in contract clauses is complete (e.g., new technology reports).
• Correctness criteria (e.g., round-off, accuracy, precision).
• Performance criteria (e.g., speed, time).
• Throughput criteria.
• System availability.
• System reliability.
• Input/output (I/O) performance.
• Memory performance.
• System communications.
• Networking capability.
• Software compatibility.
• System maintenance (component level/systems level) plan availability.
• Documentation availability.
• Readiness for the software release. ³⁷³

3.2 Acceptance Testing

Acceptance Testing is the formal testing conducted to determine whether a software system satisfies its acceptance criteria, which enables the customer to determine whether or not to accept the system. Acceptance testing is designed to determine whether the software work product is fit for use. Acceptance testing of the software work product typically forms a major portion of the acceptance plan. Once developed, the team runs the acceptance testing, commonly called a test suite, against the supplied input data and conditions. The team typically uses an acceptance test procedure to direct the software testing personnel. The software testing personnel are typically but not always independent of the project team. Software assurance personnel observe the tests. The team compares the obtained test results with the expected results. If the results match or fall within a previously agreed-to band or tolerance, the test suite is said to pass, and the work product is acceptable. If not, the work product may either be rejected or accepted on conditions previously agreed to between the customer and the software development team.

The planning for acceptance testing may consider:

• Acceptance period (is it open? is it constrained?).
• Diagnostics tests (are they part of the software requirements specification?).
• Functionality tests (what is the software work product designed to do?).
• Use of the latest production version of code (are earlier versions acceptable?).
• Differences/discrepancies (when are they acceptable? or the cause for an issue and corrective action activity?).
• Availability and readiness of the test environment.
• Availability of test and software assurance personnel to support the testing.

Acceptance testing activities include:

• Alpha testing takes place at developers' sites and involves testing the operational system by internal staff before it is released to external customers.
• Beta testing takes place at customers' sites and involves testing by a group of customers who use the system at their locations and provide feedback before the system is released to other customers. This is often called "field testing."
• System testing is invariably performed by the development team or preferably by someone independent of the developer
• User Acceptance Testing (UAT) is a process to obtain confirmation that a system meets mutually agreed-upon requirements. In software development, UAT is one of the final stages of a project and often occurs before a customer accepts the new system

Acceptance tests may also be used as regression tests before a production release. This means that new acceptance tests must be created for each iteration of the software build.

See also: SWE-193 - Acceptance Testing for Affected System and Software Behavior

 After a software work product is designed, coded, and tested against its requirements, if any deviations from the requirements and acceptance criteria still exist, they will have to be negotiated with the customer to determine if they can be accepted, or if they must be fixed before the customer accepts the product. The customer must review and agree to the acceptance test plan.

See also Topic 7.06 - Software Test Estimation and Testing Levels. 

3.3 Additional Guidance

Additional guidance related to acceptance testing may be found in the following related requirements in this handbook:

3.4 Center Process Asset Libraries

See the following link(s) in SPAN for process assets from contributing Centers (NASA Only). 

4. Small Projects

No additional guidance is available for small projects. The community of practice is encouraged to submit guidance candidates for this paragraph.

5. Resources

5.1 References

5.2 Tools

6. Lessons Learned

6.1 NASA Lessons Learned

No Lessons Learned have currently been identified for this requirement.

6.2 Other Lessons Learned

No other Lessons Learned have currently been identified for this requirement.

7. Software Assurance

7.1 Tasking for Software Assurance

7.2 Software Assurance Products

• Software Assurance acceptance criteria.
• Software Design Analysis.
• Source Code Analysis.
• Verification Activities Analysis
• Assessment of Software Reviews results
• Evidence that software acceptance criteria exist.
• Assessment that acceptance criteria for Software Engineering are reasonable per Topic 7.18 - Documentation Guidance of this Handbook, including corrective actions.

  
  

  Objective Evidence

  • Definition of the software acceptance criteria for each software product
   Definition of objective evidence

  Objective evidence is an unbiased, documented fact showing that an activity was confirmed or performed by the software assurance/safety person(s). The evidence for confirmation of the activity can take any number of different forms, depending on the activity in the task. Examples are:

  • Observations, findings, issues, risks found by the SA/safety person and may be expressed in an audit or checklist record, email, memo or entry into a tracking system (e.g. Risk Log).
  • Meeting minutes with attendance lists or SA meeting notes or assessments of the activities and recorded in the project repository.
  • Status report, email or memo containing statements that confirmation has been performed with date (a checklist of confirmations could be used to record when each confirmation has been done!).
  • Signatures on SA reviewed or witnessed products or activities, or
  • Status report, email or memo containing a short summary of information gained by performing the activity. Some examples of using a “short summary” as objective evidence of a confirmation are:
    • To confirm that: “IV&V Program Execution exists”, the summary might be: IV&V Plan is in draft state. It is expected to be complete by (some date).
    • To confirm that: “Traceability between software requirements and hazards with SW contributions exists”, the summary might be x% of the hazards with software contributions are traced to the requirements.
  • The specific products listed in the Introduction of 8.16 are also objective evidence as well as the examples listed above.

7.3 Metrics

• # of Corrective Actions (CAs) raised by SA vs. total #
  • Attributes (Type, Severity, # of days Open, Life cycle Phase Found)
  • State (Open, In work, Closed)
  • Trends of CA Open vs. Closures over time

7.4 Guidance

Step 1 Determine that the project or engineering has documented acceptance criteria for the software products.

Step 2 Assess the acceptance criteria based on guidance in the NASA Software Engineering Handbook, NASA-HDBK-2203.

Step 3 Develop acceptance criteria for all software assurance and software safety products planned to be produced for the project.

Step 4 Assess both software engineering and software assurance acceptance criteria for reasonableness.

Guidance on Perform development of acceptance criteria for all software assurance products.

Software assurance teams can better organize their development tasks if they have well-defined plans for software assurance and software safety activities. Customers and project personnel must include pre-determined criteria in these plans to evaluate software assurance and software safety work products. These criteria include clearly understanding what the customer wants the product to do and what constitutes an acceptable software assurance and software safety product. The development team documents the agreed-to acceptance criteria and the planned activities in a software assurance and software safety plan.

The software assurance and software safety team and the customer work together to ensure that they do the following:

• Identify interim and final products that will be part of acceptance activities.
• Develop the acceptance criteria and activities schedule.
• Plan how and by whom each acceptance activity will be performed.

Acceptance criteria may include:

• The product was completed successfully.
• The technical data package is current (as-built) and complete.
• If an acquisition: Technical documentation required in contract clauses is complete (e.g., new technology reports).
• Correctness criteria.
• Risk criteria.
• Reliability.
• Communications process.
• Documentation and data availability.
• Readiness for software milestones.

See also Topic 8.02 - Software Reliability

If using an Agile or incremental development process, then code verification and verification of Sprint and daily tasks need to be assured within the Sprint time frame.  At the end of the Sprint, the code and any supporting products such as documentation and test suites needed to meet the definition of “Done” as determined by the Project are saved.  Daily regression testing needs to be updated with new features and functions. The definition of “Done” for each Sprint should be chosen so that the final software system will meet the acceptance criteria

In an Agile development environment where testing is a daily activity, and the test cases, test suites, grow in step with the implementation of the story/function points, periodic examination of the tests should be performed for required reliability and safety tests as well as performance. At the end of each sprint, a working code segment is to be completed and shown to work via testing and demonstration. When sprint teams are integrating working code segments within a sprint team or among multiple sprint teams, they need to show working code not only performs but also meets required reliability, quality, and safety factors. The Software Assurance members of the sprint teams need to assure the testing is thorough and that tests are captured and that both the code and tests are configuration managed. The final integrated build of the system must still meet the acceptance criteria.

7.5 Additional Guidance

Additional guidance related to acceptance testing may be found in the following related requirements in this handbook: