1. Requirements

3.12.3 The project manager  shall define and document the acceptance criteria and conditions for the software.

1.1 Notes

NPR 7150.2, NASA Software Engineering Requirements, does not include any notes for this requirement.

1.2 Applicability Across Classes

Key:    - Applicable | - Not Applicable

2. Rationale

Software development teams can better organize their development tasks if they have well-defined plans for software activities. Customers and project personnel must include pre-determined criteria in these plans to evaluate the software work products. These criteria include clearly understanding what the customer wants the product to do and what constitutes an acceptable software product. The development team documents the agreed-to acceptance criteria and the planned activities in a software acceptance plan contained in the Software Development/Management Plan (see SDP-SMP - Software Development - Management Plan) or a separate Verification and Validation (V&V) Plan, as a formal means of guiding these activities.

3. Guidance

"Acceptance criteria" are defined as (1) those criteria that a system or component must satisfy in order to be accepted by a user, customer, or other authorized entity (ISO/IEC/IEEE 24765:2010 Systems and software engineering – Vocabulary ²³⁰), and, (2) those criteria, including performance requirements and essential conditions, which must be met before project deliverables are accepted (A Guide to the Project Management Body of Knowledge ³⁰²). The purpose of this requirement is to cause the software development team to work with the customer up front to define the criteria to be used for software acceptance. The acceptance criteria and timing can help to do the following:

• Define the software support manpower levels on a project.
• Determine the "hand over" for the software development organization to a software operation and maintenance organization.
• Help define what a contractor is to complete before flight certification can be achieved.

The software acceptance criteria definition is a key element in the overall software planning process. The software acceptance criteria need to address both software and data. If the software work product is delivered in phases, each delivery may have its own acceptance criteria.

Acceptance activities for software development begin with the planning and the development of acceptance criteria during the Formulation phase of the project. These activities and acceptance criteria can be documented in the Software Development/Management Plan (see SDP-SMP ) or a separate Software V&V Plan. As understanding of the system grows, and the requirements are better understood, the acceptance criteria can be reviewed to assure they are up to date and consistent with the system being built. They typically conclude with the system acceptance review late in the Implementation phase of the project (see the entrance and exit criteria for the Systems Acceptance Review in Topic 7.9-Entrance and Exit Criteria )

For software acquired via the acquisition process (see Topic 7.3 - Acquisition Guidance ) the acceptance criteria development follows this guidance. The final criteria must be put into the contract statement of work.

The paragraphs below deal with the acceptance criteria creation and documentation. Acceptance testing, which generates the information used to assess the satisfaction of the acceptance criteria, is also discussed.

Establish Acceptance Criteria

The software lead engineer works with the customer and the software development team to develop the appropriate acceptance plans and activities to ensure that functional requirements are properly translated into system acceptance criteria. A well-defined acceptance plan will help the software development team and the software assurance team to identify the resources required to properly support the effort. This in turn contributes to efficient and cost-effective development and V&V activities, where many of the acceptance activities occur. The team defines acceptance criteria during the Formulation phase of the software development life cycle (see SWE-019) to assure sufficient time to prepare for and perform the acceptance activities.

The software development team and the customer work together to ensure that they do the following:

• Identify interim and final products that will be part of acceptance activities.
• Develop the acceptance criteria and activities schedule.
• Plan how and by whom each acceptance activity will be performed.
• Schedule adequate time for the customer to examine and review the product.
• Prepare the acceptance plan.
• Perform formal acceptance testing at scheduled times.
• Plan the decision-making process that is based on the results of acceptance testing.

The software lead engineer works with the software team (including the software assurance personnel) to develop appropriate product reviews, tests and any audits necessary. This work includes identifying:

• The types of criteria to consider, such as customer expectations and requirements, technology limitations, environmental impact, safety, risks, total ownership and life-cycle costs, and schedule impact.
• The variations in criteria for computer software configuration items (CSCI), units, and or systems.
• The acceptable range of the criteria.
• The rank of each criterion by its importance.

Acceptance criteria may include:

• Product V&V completed successfully.
• V&V of individual products, integration of products into systems, and that system V&V has been performed or witnessed by the technical team.
• Technical data package is current (as-built) and complete.
• Transfer of certifications, warranties, or representations is complete.
• Transfer of software products, licenses, data rights, intellectual property rights, etc., is complete.
• If an acquisition: Technical documentation required in contract clauses is complete (e.g., new technology reports).
• Correctness criteria (e.g., round-off, accuracy, precision).
• Performance criteria (e.g., speed, time).
• Throughput criteria.
• System availability.
• System reliability.
• Input/output (I/O) performance.
• Memory performance.
• System communications.
• Networking capability.
• Software compatibility.
• System maintenance (component level/systems level) plan availability.
• Documentation availability.
• Readiness for software release. ³⁷³

Acceptance Testing

Acceptance Testing is the formal testing conducted to determine whether a software system satisfies its acceptance criteria, which enables the customer to determine whether or not to accept the system. Acceptance testing is designed to determine whether the software work product is fit for use. Acceptance testing of the software work product typically forms a major portion of the acceptance plan. Once developed, the team runs the acceptance testing, commonly called a test suite, against the supplied input data and conditions. The team typically uses an acceptance test procedure to direct the software testing personnel. The software testing personnel are typically but not always independent of the project team. Software assurance personnel observe the tests. The team compares the obtained test results with the expected results. If the results match, or fall within a previously agreed-to band or tolerance, the test suite is said to pass, and the work product is acceptable. If not, the work product may either be rejected or accepted on conditions previously agreed to between the customer and the software development team.

The planning for acceptance testing may consider:

• Acceptance period (is it open? is it constrained?).
• Diagnostics tests (are they part of the software requirements specification?).
• Functionality tests (what is the software work product designed to do?).
• Use of latest production version of code (are earlier versions acceptable?).
• Differences/discrepancies (when are they acceptable? or the cause for an issue and corrective action activity?).
• Availability and readiness of the test environment.
• Availability of test and software assurance personnel to support the testing.

Acceptance testing activities include:

• Alpha testing takes place at developers' sites, and involves testing of the operational system by internal staff, before it is released to external customers.
• Beta testing takes place at customers' sites, and involves testing by a group of customers who use the system at their own locations and provide feedback, before the system is released to other customers. This is often called "field testing."
• System testing is invariably performed by the development team or preferably by someone independent of the developer
• User Acceptance Testing (UAT) is a process to obtain confirmation that a system meets mutually agreed-upon requirements. In software development, UAT is one of the final stages of a project and often occurs before a customer accepts the new system

Acceptance tests may also be used as regression tests prior to a production release. This means that new acceptance tests must be created for each iteration of the software build.

 After a software work product is designed, coded, and tested against its requirements, if any deviations from the requirements and acceptance criteria still exist, they will have to be negotiated with the customer to determine if they can be accepted, or if they must be fixed prior to the customer accepting the product. The customer must review and agree to the acceptance test plan.

4. Small Projects

No additional guidance is available for small projects. The community of practice is encouraged to submit guidance candidates for this paragraph.

5. Resources

6. Lessons Learned

No lessons learned have currently been identified for this requirement.