bannerc

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Tabsetup
01. The Requirement
12. Rationale
23. Guidance
34. Small Projects
45. Resources
56. Lessons Learned
67. Software Assurance
Div
idtabs-1

1. Requirements

Excerpt

5.5.1 The project manager shall track and maintain software non-conformances (including defects in tools and appropriate ground software).  

1.1 Notes

NPR 7150.2, NASA Software Engineering Requirements, does not include any notes for this requirement.

1.2 History

Click here to view the history of this requirement: SWE-201 History

1.3 Applicability Across Classes

Applicable c
a1
b1
csc1
c1
d1
dsc1
e0
f1
g0
h0


Div
idtabs-2

2. Rationale

To make sure that all software non-conformances are addressed. Managing and tracking non-conformances, software problem reports, or software issues are critical steps to ensuring that software defects are flagged and handled properly.

Div
idtabs-3

3. Guidance

A Software non-conformance is any difference between plans, specifications, and software functionality that remains in the products after software delivery.  Even with a rigorous development process and thorough testing, some defects typically remain in the delivered products resulting in non-conformances.  

Maintaining records of non-conformances is essential to enable future software reuse or software upgrades.  Engineers who plan to reuse existing software products need thorough documentation of the root cause of anomalous behavior, documentation discrepancies, operational constraints, and operational workarounds.  Records of non-conformances can also be used to analyze the reliability and quality of delivered software products.  Non-conformance records are also useful in pinpointing specific software components or subsystems with excessive numbers of defects that remain in the delivered software.  

Software Change Tracking Tools can be used to document non-conformances, but the non-conformance data must be accessible to current and potential future users of software products.  The following is a suggested list of items that should be included in non-conformance tracking data:

  • Date non-conformance was discovered
  • How the non-conformance was discovered
  • The severity of non-conformance (Refer to SWE-202 for a discussion of Severity Levels)
  • Configuration information (i.e., Version information for the software, version information for any tools used, document revision number, etc…) 
  • Contact information for responsible engineers
  • Location of any supporting data 

Recording, tracking, and analyzing the root cause of non-conformances is a best practice.  (Refer to SWE-204 for a discussion of Root Cause Analysis.) 

Be cautious about closing overlapping software change tickets to ensure that the full scope of all associated software change tickets is addressed via retest. Be careful about missing any unique elements to the individual software change ticket.

Any aspects of control board decisions that are modified must be re-approved by the board (e.g., Impacted artifacts that need updating).

Div
idtabs-4

4. Small Projects

No additional guidance is available for small projects.

Div
idtabs-5

5. Resources

5.1 References

refstable
Show If
groupconfluence-users
Panel
titleColorred
titleVisible to editors only

Enter the necessary modifications to be made in the table below:

SWEREFs to be addedSWEREFS to be deleted


SWEREFs NOT called out in text but listed as germane: none

SWEREFs called out in text: none

5.2 Tools

Include Page
Tools Table Statement
Tools Table Statement

Div
idtabs-6

6. Lessons Learned

6.1 NASA Lessons Learned

No Lessons Learned have currently been identified for this requirement.

6.2 Other Lessons Learned

Software Change Tickets

  • Any tests (formal or informal) that fail should be rerun and verified before software change tickets are closed, in the original environment, or as close to it as possible. Preferably this would be done with the original author of the software change ticket but with appropriate control board approval.
  • Be cautious about closing overlapping software change tickets to ensure that the full scope of all associated software change tickets is addressed via retest. Be careful about missing any unique elements to the individual software change ticket.
Div
idtabs-7

7. Software Assurance

Excerpt Include
SWE-201 - Software Non-Conformances
SWE-201 - Software Non-Conformances

7.1 Tasking for Software Assurance

  1. Confirm that all software non-conformances are recorded and tracked to resolution.

  2. Confirm that accepted non-conformances include the rationale for the non-conformance.

7.2 Software Assurance Products

  • List of non-conformances for which SA has verified the closure.


    Note
    titleObjective Evidence
    • Evidence of confirmation that Tasks 1 and 2 have been completed.
    Expand
    titleDefinition of objective evidence

    Include Page
    Definition of Objective Evidence
    Definition of Objective Evidence

7.3 Metrics

  • Trends of Cybersecurity Non-Conformances over time
  • # of Non-Conformances (activities not being performed)
  • # of Non-Conformances accepted by the project
  • # of Non-Conformances (Open, Closed, Total)
  • Trends of overtimeNon-Conformances over time
  • # of Open vs. Closed Audit Non-Conformances over time
  • Trends of # of Non-Conformances from audits over time (Include counts from process and standards audits and work product audits.)
  • # of software work product Non-Conformances identified by life-cycle phase over time

7.4 Guidance

One of the key items in assuring that software quality is achieved in a system is the management of non-conformances that arise as the software system goes through the different levels of testing. Software assurance needs to keep a close watch on activities during those periods by either witnessing the tests or reviewing test reports to make sure all non-conformances (including errors, discrepancies, as well as other types of non-conformances) are being handled properly.

Activities that software assurance needs to perform associated with non-conformances:

  • Confirm all discrepancies are recorded in the discrepancy reporting system (SWE-201, Task 1)
    • When discrepancies are discovered while the test is being witnessed, check to see that the discrepancies are noted on the test results. If the discrepancy is not also recorded in the discrepancy system during the test, look at the discrepancy system after the test to verify that the discrepancy has been recorded. If the test was not witnessed, examine the resulting test results and check that any discrepancies noted are recorded also in the discrepancy system.
  • Confirm discrepancies recorded include defects in tools and ground systems (SWE-201)
    • The same process as above can be used to check those discrepancies in tools and ground systems being tested are recorded. If the discrepancies in the tools or ground systems are discovered during the testing of some other software, check to see that they are recorded in the test results where they were discovered and then verify that they have been recorded in the discrepancy system. 
  • Confirm discrepancies recorded include discrepancies received from vendors of COTS, MOTS, GOTS, OSS, reused software, as well as discrepancies from this software found during testing of the project software (as per SWE-202)
    • Review the vendor websites of the COTS, MOTS, GOTS, OSS, reused being used to get a list of the discrepancies that have been documented. Check to see that these are also listed in the project discrepancy system. For GOTS or reused software, it may be necessary to contact the developers of this software to obtain a list of the known discrepancies. Also check to see if any discrepancies in the COTS, MOTS, GOTS, OSS, or reused software are discovered during the testing of project software, verify that those discrepancies have been added to the project discrepancy list and that the vendor/developers have been notified.
  • Confirm that all recorded non-conformances have associated severity levels - correctly assigned (as per SWE-202, Task 2,3)
    • When reviewing the discrepancies documented in the project discrepancy system,  check that the appropriate severity level for each discrepancy has been recorded and that the severity level assigned meets the project-defined severity definition.
  • Confirm that each non-conformance does not exist elsewhere in the system (SWE-201, Task 2)
    • Work with the project developer to identify areas where the code may be repeated or is similar, or where a similar error in logic or design may have caused a similar discrepancy and then examine those areas to determine if the same discrepancies exist (or work with the developer to assure that he/she has checked those areas.
  • If a non-conformance is accepted for implementation of a correction, confirm that the rationale for the decision is recorded (SWE-201-Task 3)
    • Review the non-compliance records in the discrepancy system after the non-conformance has been approved for implementation and check to see that the decision rationale has been recorded.  
  • Track each non-conformance to closure
    • Periodically, review the project discrepancy system to assure that all discrepancies are being addressed in a timely manner. Discrepancies approved for implementation must be actually implemented and then tested, including regression testing. Other discrepancies should eventually have some final resolution like a work-around or a decision that the discrepancy doesn't need to be fixed.
  • Review metrics on the non-conformances, including:
    • Number of non-conformances at each severity level (SWE-202)
    • Total number of non-conformances/discrepancies (SWE-201)
    • Number of non-conformances/discrepancies closed (SWE-201)
    • Number of non-conformances/discrepancies open during this period (SWE-201)
    • Age of non-conformance/discrepancy (SWE-201)
    • The severity of each non-conformance/discrepancy (SWE-201)
  • Be cautious about closing overlapping software change tickets to ensure that the full scope of all associated software change tickets is addressed via retest. Be careful about missing any unique elements to the individual software change ticket.

  • Any aspects of control board decisions that are modified must be re-approved by the board (e.g., Impacted artifacts that need updating).