bannera

Book A.
Introduction

Book B.
7150 Requirements Guidance

Book C.
Topics

Tools,
References, & Terms

SPAN
(NASA Only)

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Migration of unmigrated content due to installation of a new plugin


{alias:SWE-034} {tabsetup:1. The Requirement|2. Rationale|3. Guidance|4. Small Projects|5. Resources|6. Lessons Learned} {div3:id=tabs-1} h1. 1. Requirements
Wiki Markup
Tabsetup
1. The Requirement
1. The Requirement
12. Rationale
23. Guidance
34. Small Projects
45. Resources
56. Lessons Learned


Div
idtabs-1

1. Requirements

2.5.3

The

project

shall

define

and

document

or

record

the

acceptance

criteria

and

conditions

for

the

software.

h2. {color:#003366}{*}

1.1

Notes{*}{color} NPR

Notes

NPR 7150.2

does not include any notes for this requirement. h2.

, NASA Software Engineering Requirements, does not include any notes for this requirement.

1.2

Applicability

Across

Classes

{applicable:asc=1|ansc=1|bsc=1|bnsc=1|csc=1|cnsc=1|dsc=1|dnsc=1|esc=1|ensc=0|f=1|g=1|h=0} {div3} {div3:id=tabs-2} h1. 2. Rationale Software development teams can better organize their development tasks if they have well defined plans for software activities. Customers and project personnel must include pre-determined criteria in these plans to evaluate the software work products. These criteria include clearly understanding what the customer wants the product to do and what constitutes an acceptable software product. The development team should document the agreed-to acceptance criteria and the planned activities in a software acceptance plan contained in the Software Development/Management Plan (see [SWE-102|SWE-102]) or a separate Verification and Validation Plan, as a formal means of guiding these activities. {div3} {div3:id=tabs-3} h1. 3. Guidance 'Acceptance criteria' are defined as (1) those criteria that a system or component must satisfy in order to be accepted by a user, customer, or other authorized entity ([ISO/IEC/IEEE 24765:2010 Systems and software engineering -- Vocabulary|http://www.iso.org/iso/catalogue_detail.htm?csnumber=50518]), and, (2) those criteria, including performance requirements and essential conditions, which must be met before project deliverables are accepted [(A Guide to the Project Management Body of Knowledge (PMBOK® Guide) -- Fourth Edition|http://marketplace.pmi.org/Pages/ProductDetail.aspx?GMProduct=00101095501]). The purpose of this requirement is to cause the software development team to work with the customer up front to define the criteria to be used for software acceptance. The acceptance criteria and timing can help * define the software support manpower levels on a project * determine the 'hand over' for the software development organization to a software operation and maintenance organization, * help define what a contractor is to complete before flight certification can be achieved The software acceptance criteria definition is a key element in the overall software planning process. The software acceptance criteria need to address both software and data. If the software work product is delivered in phases, each delivery may have its own acceptance criteria. Acceptance activities for software development begin with the planning and the development of acceptance criteria during the Formulation phase of the project. These activities and acceptance criteria can be documented in the Software Development/Management Plan (see [SWE-102|SWE-102] ) or a separate Software Verification and Validation Plan. As understanding of the system grows, and the requirements are beter understood, the acceptance criteria can be reviewed to assure they are up to date and consistent with the system being built. They typically conclude with the system acceptance review late in the Implementation phase of the project (see the entrance and exit criteria for the Systems Acceptance Review in [Topic 7.3|http://nasa7150.onconfluence.com/display/7150/7.3+-+Entrance+and+Exit+Criteria] ) For software acquired via the acquisition process (see [Topic 7.7|http://nasa7150.onconfluence.com/display/7150/7.7+-+Acquisition+Guidance] ) the acceptance criteria development should follow this guidance. The final criteria must be put into the contract statement of work. The paragraphs below deal with the acceptance criteria creation and documentation. Acceptance testing, which generates the information used to assess the satisfaction of the acceptance criteria, is also discussed. *Establish Acceptance Criteria* The software lead engineer works with the customer and the software development team to develop the appropriate acceptance plans and activities to ensure that functional requirements are properly translated into system acceptance criteria. A well-defined acceptance plan will help the software development team and the software assurance team to identify the resources required to properly support the effort. This in turn contributes to efficient and cost-effective development and verification and validation activities, where many of the acceptance activities occur. The team should define acceptance criteria during the Formulation phase of the software development life cycle (see [SWE-019|SWE-019]) to assure sufficient time to prepare for and perform the acceptance activities. The software development team and the customer should work together and make sure that they: * Identify interim and final products that will be part of acceptance activities * Develop the acceptance criteria and activities schedule * Plan how and by whom each acceptance activity will be performed * Schedule adequate time for the customer to examine and review the product * Prepare the acceptance plan * Perform formal acceptance testing at scheduled times * Plan the decision-making process that is based on the results of acceptance testing The software lead engineer works with the software team (including the software assurance personnel) to develop appropriate product reviews, tests and any audits necessary. This work includes identifying: * The types of criteria to consider, such as customer expectations and requirements, technology limitations, environmental impact, safety, risks, total ownership and life-cycle costs, and schedule impact * The variations in criteria for {term:CSCI}'s, units, and or systems * The acceptable range of the criteria * The rank of each criterion by its importance Acceptance criteria may include: * Product verification and validation completed successfully * Verification and validation of individual products, integration of products into systems, and that system verification and validation has been performed or witnessed by the technical team * Technical data package is current (as-built) and complete * Transfer of certifications, warranties, or representations is complete * Transfer of software products, licenses, data rights, intellectual property rights, etc., is complete * (If an acquisition) Technical documentation required in contract clauses is complete (e.g., new technology reports) * Correctness criteria ( e.g., round-off, accuracy, precision) * Performance criteria (e.g., speed, time) * Throughput criteria * System availability * System reliability * I/O performance * Memory performance * System communications * Networking capability * Software compatibility * System maintenance (component level/systems level)plan availability * Documentation availability * Readiness for software release ^3^ *Acceptance Testing* Acceptance Testing is the formal testing conducted to determine whether a software system satisfies its acceptance criteria, which enables the customer to determine whether or not to accept the system. Acceptance testing is designed to determine whether the software work product is fit for use. Acceptance testing of the software work product typically forms a major portion of the acceptance plan. Once developed, the team runs the acceptance testing, commonly called a test suite, against the supplied input data and conditions. The team typically uses an acceptance test procedure to direct the software testing personnel. The software testing personnel are typically but not always independent of the project team. Software assurance personnel observe the tests. The team compares the obtained test results with the expected results. If the results match, or fall within a previously agreed-to band or tolerance, the test suite is said to pass, and the work product is acceptable. If not, the work product may either be rejected or accepted on conditions previously agreed to between the customer and the software development team. The planning for acceptance testing should consider: * Acceptance period (is it open, is it constrained?) * Diagnostics tests (are they part of the software requirements specification?) * Functionality tests (what is the software work product designed to do?) * Use of latest production version of code (are earlier versions acceptable?) * Differences/discrepancies (when are they acceptable or the cause for an issue and corrective action activity?) * Availability and readiness of the test environment * Availability of test and software assurance personnel to support the testing Acceptance testing activities include: * Alpha testing takes place at developers' sites, and involves testing of the operational system by internal staff, before it is released to external customers * Beta testing takes place at customers' sites, and involves testing by a group of customers who use the system at their own locations and provide feedback, before the system is released to other customers. This is often called "field testing" * System testing is invariably performed by the development team or preferably by someone independent of the developer * User Acceptance Testing (UAT) is a process to obtain confirmation that a system meets mutually agreed-upon requirements. In software development, UAT is one of the final stages of a project and often occurs before a customer accepts the new system Acceptance tests may also be used as regression tests prior to a production release. This means that new acceptance tests must be created for each iteration of the software build. {panel}Criteria for acceptance testing are often documented in the verification matrix of the software verification plans. (See [SWE-028|SWE-028] ). Verification results that are used in software acceptance reviews are typically documented in software verification reports. (See [SWE-030|SWE-030] ). Test results for software work products subjected to acceptance testing must be documented in a test report or an acceptance data package. (See [SWE-118|SWE-118] and [SWE-119|SWE-119] for related information.) {panel} After a software work product is designed, coded, and tested against its requirements, if any deviations from the requirements and acceptance criteria still exist, they will have to be negotiated with the customer to determine if they can be accepted, or if they must be fixed prior to the customer accepting the product. The customer must review and agree to the acceptance test plan. {panel}The entrance criteria and the exit (success) criteria should be developed and documented during the acceptance planning activities (see [Topic 7.3|http://nasa7150.onconfluence.com/display/7150/7.3+-+Entrance+and+Exit+Criteria] ) for the System Acceptance Review. {panel} {div3} {div3:id=tabs-4} h1. 4. Small Projects There is no information applicable to this section. {div3} {div3:id=tabs-5} h1. 5. Resources # [NASA Systems Engineering Handbook|http://education.ksc.nasa.gov/esmdspacegrant/Documents/NASA%20SP-2007-6105%20Rev%201%20Final%2031Dec2007.pdf], NASA/SP-2007-6105 Rev1, 2007 # IEEE Computer Society, "[IEEE Standard for Software Reviews and Audits",|http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=4601584&tag=1] IEEE Std 1028-2008, 2008. # [Release of NASA Software|http://nodis3.gsfc.nasa.gov/displayDir.cfm?t=NPR&c=2210&s=1C], NPR 2210.1C, 2010 # [NASA Software Assurance Standard|https://standards.nasa.gov/documents/detail/3315130], NASA STD 8739.8, 2005 # [ISO/IEC/IEEE 24765:2010 Systems and software engineering -- Vocabulary|http://www.iso.org/iso/catalogue_detail.htm?csnumber=50518] # [(A Guide to the Project Management Body of Knowledge (PMBOK® Guide) -- Fourth Edition|http://marketplace.pmi.org/Pages/ProductDetail.aspx?GMProduct=00101095501] {Toolstable} {div3} {div3:id=tabs-6} h2. 6. Lessons Learned No lessons learned have currently been identified for this requirement. {div3} {tabclose}


applicable
f1
g1
h0
ansc1
asc1
bnsc1
csc1
bsc1
esc1
cnsc1
dnsc1
dsc1
ensc0



Div
idtabs-2

2. Rationale

Software development teams can better organize their development tasks if they have well-defined plans for software activities. Customers and project personnel must include pre-determined criteria in these plans to evaluate the software work products. These criteria include clearly understanding what the customer wants the product to do and what constitutes an acceptable software product. The development team documents the agreed-to acceptance criteria and the planned activities in a software acceptance plan contained in the Software Development/Management Plan (see SWE-102) or a separate Verification and Validation (V&V) Plan, as a formal means of guiding these activities.


Div
idtabs-3

3. Guidance

"Acceptance criteria" are defined as (1) those criteria that a system or component must satisfy in order to be accepted by a user, customer, or other authorized entity (ISO/IEC/IEEE 24765:2010 Systems and software engineering – Vocabulary

sweref
230
230
), and, (2) those criteria, including performance requirements and essential conditions, which must be met before project deliverables are accepted (A Guide to the Project Management Body of Knowledge
sweref
302
302
). The purpose of this requirement is to cause the software development team to work with the customer up front to define the criteria to be used for software acceptance. The acceptance criteria and timing can help to do the following:

  • Define the software support manpower levels on a project.
  • Determine the "hand over" for the software development organization to a software operation and maintenance organization.
  • Help define what a contractor is to complete before flight certification can be achieved.

The software acceptance criteria definition is a key element in the overall software planning process. The software acceptance criteria need to address both software and data. If the software work product is delivered in phases, each delivery may have its own acceptance criteria.

Acceptance activities for software development begin with the planning and the development of acceptance criteria during the Formulation phase of the project. These activities and acceptance criteria can be documented in the Software Development/Management Plan (see SWE-102 ) or a separate Software V&V Plan. As understanding of the system grows, and the requirements are better understood, the acceptance criteria can be reviewed to assure they are up to date and consistent with the system being built. They typically conclude with the system acceptance review late in the Implementation phase of the project (see the entrance and exit criteria for the Systems Acceptance Review in Topic 7.9-Entrance and Exit Criteria )

For software acquired via the acquisition process (see Topic 7.3 - Acquisition Guidance ) the acceptance criteria development follows this guidance. The final criteria must be put into the contract statement of work.

The paragraphs below deal with the acceptance criteria creation and documentation. Acceptance testing, which generates the information used to assess the satisfaction of the acceptance criteria, is also discussed.

Establish Acceptance Criteria

The software lead engineer works with the customer and the software development team to develop the appropriate acceptance plans and activities to ensure that functional requirements are properly translated into system acceptance criteria. A well-defined acceptance plan will help the software development team and the software assurance team to identify the resources required to properly support the effort. This in turn contributes to efficient and cost-effective development and V&V activities, where many of the acceptance activities occur. The team defines acceptance criteria during the Formulation phase of the software development life cycle (see SWE-019) to assure sufficient time to prepare for and perform the acceptance activities.

The software development team and the customer work together to ensure that they do the following:

  • Identify interim and final products that will be part of acceptance activities.
  • Develop the acceptance criteria and activities schedule.
  • Plan how and by whom each acceptance activity will be performed.
  • Schedule adequate time for the customer to examine and review the product.
  • Prepare the acceptance plan.
  • Perform formal acceptance testing at scheduled times.
  • Plan the decision-making process that is based on the results of acceptance testing.

The software lead engineer works with the software team (including the software assurance personnel) to develop appropriate product reviews, tests and any audits necessary. This work includes identifying:

  • The types of criteria to consider, such as customer expectations and requirements, technology limitations, environmental impact, safety, risks, total ownership and life-cycle costs, and schedule impact.
  • The variations in criteria for CSCI's (Computer Software Configuration Items), units, and or systems.
  • The acceptable range of the criteria.
  • The rank of each criterion by its importance.

Acceptance criteria may include:

  • Product V&V completed successfully.
  • V&V of individual products, integration of products into systems, and that system V&V has been performed or witnessed by the technical team.
  • Technical data package is current (as-built) and complete.
  • Transfer of certifications, warranties, or representations is complete.
  • Transfer of software products, licenses, data rights, intellectual property rights, etc., is complete.
  • If an acquisition: Technical documentation required in contract clauses is complete (e.g., new technology reports).
  • Correctness criteria ( e.g., round-off, accuracy, precision).
  • Performance criteria (e.g., speed, time).
  • Throughput criteria.
  • System availability.
  • System reliability.
  • Input/output (I/O) performance.
  • Memory performance.
  • System communications.
  • Networking capability.
  • Software compatibility.
  • System maintenance (component level/systems level) plan availability.
  • Documentation availability.
  • Readiness for software release.
    sweref
    373
    373

Acceptance Testing

Acceptance Testing is the formal testing conducted to determine whether a software system satisfies its acceptance criteria, which enables the customer to determine whether or not to accept the system. Acceptance testing is designed to determine whether the software work product is fit for use. Acceptance testing of the software work product typically forms a major portion of the acceptance plan. Once developed, the team runs the acceptance testing, commonly called a test suite, against the supplied input data and conditions. The team typically uses an acceptance test procedure to direct the software testing personnel. The software testing personnel are typically but not always independent of the project team. Software assurance personnel observe the tests. The team compares the obtained test results with the expected results. If the results match, or fall within a previously agreed-to band or tolerance, the test suite is said to pass, and the work product is acceptable. If not, the work product may either be rejected or accepted on conditions previously agreed to between the customer and the software development team.

The planning for acceptance testing may consider:

  • Acceptance period (is it open? is it constrained?).
  • Diagnostics tests (are they part of the software requirements specification?).
  • Functionality tests (what is the software work product designed to do?).
  • Use of latest production version of code (are earlier versions acceptable?).
  • Differences/discrepancies (when are they acceptable? or the cause for an issue and corrective action activity?).
  • Availability and readiness of the test environment.
  • Availability of test and software assurance personnel to support the testing.

Acceptance testing activities include:

  • Alpha testing takes place at developers' sites, and involves testing of the operational system by internal staff, before it is released to external customers.
  • Beta testing takes place at customers' sites, and involves testing by a group of customers who use the system at their own locations and provide feedback, before the system is released to other customers. This is often called "field testing."
  • System testing is invariably performed by the development team or preferably by someone independent of the developer
  • User Acceptance Testing (UAT) is a process to obtain confirmation that a system meets mutually agreed-upon requirements. In software development, UAT is one of the final stages of a project and often occurs before a customer accepts the new system

Acceptance tests may also be used as regression tests prior to a production release. This means that new acceptance tests must be created for each iteration of the software build.


Panel

Criteria for acceptance testing are often documented in the verification matrix of the software verification plans. (See SWE-028.) Verification results that are used in software acceptance reviews are typically documented in software verification reports. (See SWE-030.) Test results for software work products subjected to acceptance testing must be documented in a test report or an acceptance data package. (See SWE-118 and SWE-119 for related information.)


After a software work product is designed, coded, and tested against its requirements, if any deviations from the requirements and acceptance criteria still exist, they will have to be negotiated with the customer to determine if they can be accepted, or if they must be fixed prior to the customer accepting the product. The customer must review and agree to the acceptance test plan.


Panel

The entrance criteria and the exit (success) criteria are developed and documented during the acceptance planning activities (see Topic 7.9 ) for the System Acceptance Review.



Div
idtabs-4

4. Small Projects

No additional guidance is available for small projects. The community of practice is encouraged to submit guidance candidates for this paragraph.


Div
idtabs-5

5. Resources


refstable



Toolstable


Div
idtabs-6

6. Lessons Learned

No lessons learned have currently been identified for this requirement.