This version of SWEHB is associated with NPR 7150.2B. Click for the latest version of the SWEHB based on NPR7150.2C
4.4.5 The project manager shall unit test the software code per the plans for software testing.
NPR 7150.2, NASA Software Engineering Requirements, does not include any notes for this requirement.
1.2 Applicability Across Classes
Classes F and G are labeled with "X (not OTS)." This means that this requirement does not apply to off-the-shelf software for these classes.
Class A B C CSC D DSC E F G H Applicable?
Key: - Applicable | - Not Applicable
A & B = Always Safety Critical; C & D = Not Safety Critical; CSC & DSC = Safety Critical; E - H = Never Safety Critical.
Unit testing is the process for testing the range of inputs to a unit to ensure that only the intended outputs are produced. By doing this at the lowest level, fewer issues will be discovered when the components are later integrated and tested as a whole. Therefore, during unit testing, it is important to check the maximum and minimum values, invalid values, empty and corrupt data, etc. for each input and output to ensure the unit properly handles the data (processes or rejects it).
Ensuring that developers perform unit testing in accordance with written test plans helps build quality into the software from the beginning and allows bugs to be corrected early in the project life cycle when such corrections cost the least to the project.
Per IEEE STD 610.12-1990, IEEE Standard Glossary of Software Engineering Terminology, a "unit" is defined as:
(1) A separately testable element specified in the design of a computer software component.
(2) A logically separable part of a computer program.
(3) A software component that is not subdivided into other components.
Given the low-level nature of a unit of code, the person most able to fully test that unit is the developer who created it.
Projects ensure that the appropriate test environment, test materials, and personnel training (SWE-017), are in place and then conduct unit tests per the approved plans (Software Test Plan), according to the schedule (SWE-016), and with proper monitoring per the software assurance plan, making sure that:
- Criteria for a successful test are established prior to the test.
- Test environment represents inputs, output, and stimulus the unit will experience in operation.
- Capture weaknesses or differences between unit test environment and actual target environment.
- In accordance with the approved plans for unit testing:
- Unit test results are captured.
- Issues are identified and documented (some minor issues, such as typos, as defined by the project, may simply be corrected without documentation).
- Unit test issues are corrected; these may include:
- Issues found in the code.
- Issues found in test instruments (e.g., scripts, data, procedures).
- Issues found in testing tools (e.g., setup, configuration).
- Unit test corrections are captured (for root cause analysis, as well as proof that the unit test plans were followed).
- Unit test results are evaluated by someone other than the tester to confirm the results, as applicable and practical; evaluation results captured.
- Unit test data, scripts, test cases, procedures, test drivers, test stubs are captured for reference and any required regression testing.
- Notes captured in software engineering notebooks or other documents are captured for reference.
- Objective evidence that unit tests were completed and unit test objectives met is captured in the Software Development Folders (SDFs) or other appropriate project location as called out in the project documentation (e.g., Software Development Plan (SDP)/ Software Management Plan (SMP), Configuration Management (CM)Plan). 013
- Unit test metrics captured, as appropriate and defined for the project.
Per NASA-STD-8719.13, NASA Software Safety Standard; and NASA-GB-8719.13, NASA Software Safety Guidebook, software assurance is to "Verify unit testing and data verification is completed before the unit is integrated." 271 Either software assurance or Independent Verification and Validation (IV&V) personnel "Verify unit tests adequately test the software and are actually performed." 276 When less formal confirmation of unit testing is needed, a software team lead or other designated project member may verify completeness and correctness of the testing by comparing the results to the test plan to ensure that all logic paths have been tested and verifying the test results are accurate. 047
Unit testing tools and some integrated development environments (IDEs) can auto-generate unit tests based on the code. These tools provide a quick method to generate unit tests, but may not completely exercise the unit of code. Rerun units tests each time the unit is updated to ensure the code continues to work as expected. When continuous integration is part of the life cycle, all of the unit tests are rerun each time the code is updated to ensure only working code is integrated.
Documented test results, results evaluations, issues, problem reports, corrections, and tester notes can all serve as evidence that unit tests were completed. Comparing those documents to the software test plans for unit testing can ensure the tests were completed in accordance with those documented procedures.
Make sure evidence of all test passes is captured.
NASA-GB-8719.13, NASA Software Safety Guidebook, 276 further states in the section on safety-critical unit test plans that "documentation is required to prove adequate safety testing of the software." Therefore, unit test results can play an important role in supporting reviews of safety-critical software.
Consult Center PALs for Center-specific guidance and resources related to unit testing.
Additional guidance related to unit testing may be found in the following related requirements in this handbook:
4. Small Projects
Projects with limited budgets and personnel may choose to perform unit testing or capture unit test results and artifacts in a less formal manner than projects with greater resources. Regardless of the formality of the procedures used, the software test plans for unit testing need to describe the test environment/setup, results capture, simple documentation procedures, and compliance checks against the procedures. Some Centers have tailored lean unit test procedures and support tools specifically for small projects.
Tools relative to this SWE may be found in the table below. You may wish to reference the Tools Table in this handbook for an evolving list of these and other tools in use at NASA. Note that this table should not be considered all-inclusive, nor is it an endorsement of any particular tool. Check with your Center to see what tools are available to facilitate compliance with this requirement.
"Based on the Eclipse platform, Wind River Workbench is a collection of tools that accelerates time-to-market for developers building devices with VxWorks and Wind River Linux. Through tight integration with the industry's leading RTOS and the leading device Linux distribution, Workbench offers the only end-to-end, open standards-based collection of tools for device software design, development, debugging, test, and management."
IV&V Centers?, ARC (no version noted)
CxxTest is a unit testing framework for C++ that is similar in spirit to Junit, CppUnit, and xUnit. CxxTest is easy to use because it does not require precompiling a CxxTest testing library. It employs no advanced features of C++ and it supports a very flexible form of test discovery. Version 4.4
6. Lessons Learned
The NASA Lessons Learned database contains the following lessons learned related to unit testing:
- MPL Uplink Loss Timer Software/Test Errors (1998) (Plan to test against full range of parameters.) Lesson Number 0939: Lesson Learned No. 2 states: "Unit and integration testing should, at a minimum, test against the full operational range of parameters. When changes are made to database parameters that affect logic decisions, the logic should be re-tested." 530
- Computer Software/Configuration Control/Verification and Validation (V&V) (Unit level V&V needed for auto-code and auto-code generators.) Lesson Number 1023: "The use of the Matrix X autocode generator for ISS software can lead to serious problems if the generated code and Matrix X itself are not subjected to effective configuration control or the products are not subjected to unit-level V&V. These problems can be exacerbated if the code generated by Matrix X is modified by hand." 533