1. Multi-Logic Testing

1.1 – Definitions

• Unit Testing
  • Testing of individual routines and modules by the developer or an independent tester (ISO/IEC/IEEE 24765).
  • A test of individual programs or modules in order to ensure that there are no analysis or programming errors (ISO/IEC 2382-20).
  • Test of individual hardware or software units or groups of related units. (ISO/IEC/IEEE 24765)
    
    
• Software Verification
  • Confirmation that products properly reflect the requirements specified for them. Verification ensures that “you built it right.” (Source: IEEE 1012)

• Software Integration Testing -
• Integration testing examines if the individual components developed and tested separately, interact as expected when they are combined to form a part of a larger system. (Source: IEEE)

• Integration Testing -
  • Integration testing examines if the individual components developed and tested separately, interact as expected when they are combined to form a part of a larger system. (Source: IEEE)

1.2 Multi-Logic Condition Requirements

• Simplified Example of Multi-logic Condition Software Requirement:
  • “ If mode is STANDBY and status is (ENABLED or ACTIVE), the software shall store the current time.”
    
    
• For software requirements, there should be very few multi-logic condition requirements
  • Multi-logic condition requirements complicate testing and should be reduced or eliminated, if possible
• Operators used in textual requirements are typically “and” or “or”
  
  
• The example above can be restated as: If A and (B or C) then store the current time.
  • If the result of “A and (B or C)” is TRUE, then the current time will be stored.
  • Operators are “and” and “or”, operands are “A” , “B”, and “C” each with two possible states (i.e. True/False)
  • Three operands yields a possible 2n test combinations, 8 possible test cases

1.3 Verifying Multi-logic Condition Requirements

 1. If possible, rewrite software requirement to eliminate multi-logic conditions

 2. The preferred method of verification should be test

• Inspection, Analysis, or Demonstration should only be used if test is not possible

 3. Run a test case for each possible combination of operand values (2n)

• Test all possible permutations (i.e. cover the full truth table)
• Note this method is analogous to the Multiple Condition Coverage (MCC) test method used for code unit testing

 4. If running all 2n test cases is not possible (due to hardware limitations or resource constraints):

• At a minimum, run the subset of tests to prove that varying each operand affects the result, independently
  • Must perform an analysis to determine the correct subset of tests
  • Note this method is analogous to the Modified Condition/Decision Coverage (MCDC) test method used for code unit testing

Example on the following tabs illustrate application of this recommendation. 

 1.4 Determining the Minimum Set of Tests Required

 2. Recommendations for Requirements When Conditions Are Not Discrete

Simplified Example:

• “ If mode is PROCESSING and TEMPERATURE is greater than 50.0, the software shall set a WARNING.”

The example above can be restated as: If A and (B>50.0) then set a warning.

• In this case, operand A can have two possible states but B can have many states
• Very difficult to test and analyze all combinations

3. Summary - Verifying Multi-logic Condition Requirements

• Recommend running the full set of 2ⁿ test cases for Multi-condition requirements
  • For safety critical requirements, must perform the 2ⁿ test cases or request a waiver (to NPR 7150.2 

    Swerefn
    refnum 083

    and NASA-STD-8739.8A

    Swerefn
    refnum 278

    )
  • Also recommend required input from Subject Matter Experts/Responsible Engineers to ensure tests adequately cover intent of requirement.
    
    
• If running the full set of test is not practical, run the minimum subset of tests to prove that varying each operand affects the result, independently
  
• Why run the full set of 2ⁿ test cases instead of the n+1 set?
  • Determining Independent outcome is implementation dependent
    • Therefore, we need to decouple requirements testing from implementation
    • Only way that can be assured is by running all permutations
  • Intensive to analyze and prone to error, and should be decoupled anyway
  • No automated tools for determining which cases to remove based on requirements
  • Relatively low number of these type requirements should exist
    • Should not require many additional test cases
      
      
• For requirements where conditions are not discrete, follow best practice and run tests with values just below the limit, at the limit, and just above the limit.
  • Recommend thorough requirements analysis to ensure all permutations are represented by selected input values.
    
    
• For requirements where conditions have a range of values, follow best practice and run tests with values just below the limit, at the limit, just above the limit, and at least one value in the middle of the range.

See SWE-134 - Safety Critical Software Design Requirements for additional guidance. 

4. Resources

4.1 References

4.2 Tools

5. Lessons Learned

Currently there are no Lessons Learned identified for this topic.