- 1. The Requirement
- 2. Rationale
- 3. Guidance
- 4. Small Projects
- 5. Resources
- 6. Lessons Learned
- 7. Software Assurance
5.5.2 The project manager shall define and implement clear software severity levels for all software non-conformances (including tools, COTS, GOTS, MOTS, OSS, reused software components, and applicable ground systems).
At a minimum, classes should include loss of life or loss of vehicle, mission success, visible to the user with operational workarounds, and an ‘other’ class that does not meet previous criteria.
Click here to view the history of this requirement: SWE-202 History
1.3 Applicability Across Classes
Key: - Applicable | - Not Applicable
A & B = Always Safety Critical; C & D = Sometimes Safety Critical; E - F = Never Safety Critical.
Severity is defined as the degree of impact a defect has on the development or operation of the software being tested. A higher effect on the system functionality will lead to the assignment of higher severity to the bug. Severity indicates the seriousness of the defect on the software functionality. These software severity levels should be defined and implemented clearly.
Assigning software severity levels for software non-conformances allows engineers to focus their efforts on problems with the highest potential impact. Severity levels are different for each project based on the criticality of the system to NASA’s major programs and projects; potential risk to astronauts, employees or the public; and magnitude of the potential loss of government investment. Other project unique characteristics can also affect severity levels of particular non-conformances (e.g., non-conformances that disrupt science data operations, causing missed science opportunities). Project Managers should enlist software engineers, systems engineers, subsystem engineers, and other project personnel to help define severity levels for a particular project based on the project's unique characteristics. This same set of personnel should also participate, as applicable, in reviewing assigned severity levels to non-conformances that are discovered.
Severity levels are particularly important for non-conformances discovered in COTS, GOTS, MOTS, OSS, and reused software components. Most COTS, OSS and some GOTS, MOTS and reused software components share lists of known defects and non-conformances. The intent of the requirement is for the user of the software to verify if a site or list of non-conformances or reported bugs is maintained by the developing organization and to review the list of known non-conformances or report bugs to see if the non-conformances or reported bugs could or do impact the software component. Most commercial products and open-source software have a site that shows a list of well know bugs or non-conformances. The requirement is to research the information and see if any of the known bugs impact the software component being used by the project. Non-conformance tracking data can be used to assess the quality and determine the latent risk to the system inherent in using COTS, GOTS, MOTS, OSS, and reused software components.
At a minimum, software non-conformance severity levels should be defined as Major (High) or Minor (Low). Additionally, levels such as Trivial or Editorial can be used for items that do not impact normal operations. More severe non-conformances can be classified as Critical or Unacceptable. As stated earlier, these severity levels are project-specific and should be defined and documented. But keeping the number of definitions to a minimum will simplify tracking and reporting metrics.
The following example of severity definitions is used by one organization at NASA:
P1 - Critical: A critical priority change request is considered to be imperative to the success of the project, and likewise, may have a detrimental impact on the project if not addressed promptly. This type of change request is mandatory and must be completed. The timeframe for estimating the effort and cost required to implement a critical change request should be one (1) week or less. Examples of critical change requests are legal mandates, functionality to meet core business process requirements, or data integrity with respect to database content.
P2 - High: A high priority change request is considered to be important to the success of the project. The timeframe for estimating the effort and cost required to implement a high priority change request should be two (2) to four (4) weeks. Examples of high priority change requests are issues and problems resulting from data integrity, legal mandates, and add-ons to improve data quality.
P3 - Medium: A medium priority change request has the potential to impact the successful completion of the project but is not an immediate help nor hindrance. The timeframe for estimating the effort and cost required to implement a medium priority change request should be four (4) to six (6) weeks. Examples of medium priority change requests are requests that improve workflow processes.
P4 - Low: Low priority change requests need to be addressed if the time and budget permit. Low priority changes requests are managed, as resources are available. The timeframe guideline for estimating the effort and cost required to implement a low priority change request is more than six (6) weeks. Examples of low priority change requests are cosmetic changes or “fixes” that do not affect business functional requirements or deliverables.
4. Small Projects
No additional guidance is available for small projects.
No references have been currently identified for this SWE. If you wish to suggest a reference, please leave a comment below.
6. Lessons Learned
6.1 NASA Lessons Learned
No Lessons Learned have currently been identified for this requirement.
6.2 Other Lessons Learned
No other Lessons Learned have currently been identified for this requirement.
7. Software Assurance
7.1 Tasking for Software Assurance
Confirm that all software non-conformances severity levels are defined.
Assess the application and accuracy of the defined severity levels to software non-conformances.
Confirm that the project assigns severity levels to non-conformances associated with tools, Commerical-Off-The-Shelf (COTS), Government-Off-The-Shelf (GOTS), Modified-Off-The-Shelf (MOTS), Open Source Software (OSS), reused software components, and applicable ground systems.
Maintain or have access to the number of software non-conformances at each severity level for each software configuration item.
7.2 Software Assurance Products
- Assessment of Accuracy of Severity-Level Application to Non-conformances
- SA assessment of severity levels assigned to software non-conformances, including risks and issues
- Information for accessing number of software non-conformances at each severity level for each software configuration item.
Definition of objective evidence
- Evidence of confirmation that Tasks 1 and 2 have been completed.
Objective evidence is an unbiased, documented fact showing that an activity was confirmed or performed by the software assurance/safety person(s). The evidence for confirmation of the activity can take any number of different forms, depending on the activity in the task. Examples are:
- Observations, findings, issues, risks found by the SA/safety person and may be expressed in an audit or checklist record, email, memo or entry into a tracking system (e.g. Risk Log).
- Meeting minutes with attendance lists or SA meeting notes or assessments of the activities and recorded in the project repository.
- Status report, email or memo containing statements that confirmation has been performed with date (a checklist of confirmations could be used to record when each confirmation has been done!).
- Signatures on SA reviewed or witnessed products or activities, or
- Status report, email or memo containing Short summary of information gained by performing the activity. Some examples of using a “short summary” as objective evidence of a confirmation are:
- To confirm that: “IV&V Program Execution exists”, the summary might be: IV&V Plan is in draft state. It is expected to be complete by (some date).
- To confirm that: “Traceability between software requirements and hazards with SW contributions exists”, the summary might be x% of the hazards with software contributions are traced to the requirements.
- Total # of Non-Conformances over time (Open, Closed, # of days Open, and Severity of Open)
- # of Non-Conformances in current reporting period (Open, Closed, Severity)
- # of Non-Conformances identified in embedded COTS, GOT, MOTS, OSS, or reused components in ground or flight software vs. # of Non-Conformances successfully closed
- # of Non-Conformances identified in source code products used (Open, Closed)
- # of software Non-Conformances at each Severity level for each software configuration item (Open, Closed)
Software assurance confirms that the project has defined and documented severity levels for non-compliances before the start of any testing. Typically, NASA severity levels are from 1 to 5 or 1 to 4 with 1 being the most severe. Some example definitions are found in the software requirement guidance for SWE-202 (this requirement). The project definitions of the severity levels can be documented in any of several places. In many cases, they may be documented in either the test plans or in the test procedures. Often a project will build the definitions of the severity levels in the discrepancy tracking tool being used to track the errors found during testing. Severity levels may not be used officially for unit testing, but even there it is important to know which errors are the most critical and which should be fixed first. Severity levels should be used for all other levels of testing.
Once the project begins testing, software assurance needs to be monitoring the activity by either witnessing tests or reviewing test report documentation. Check to be sure that all the non-conformances that are found are documented in the project discrepancy report system along with the severity level of the non-conformance. Review the assigned severity level to ensure that severity levels are being assigned as per the project definitions of the categories of non-conformances.
Software assurance also needs to review the developer maintained lists of non-conformances found or reported in COTS, MOTS, GOTS, OSS, and other reused software. These non-conformances should have been assigned the appropriate severity levels that match the project definitions for non-conformances occurring in the project software. (In other words, don’t just assume the severity levels recorded in received lists of reused software will reflect the same severity level when in use in the project software).
Software assurance confirms that the project metrics are kept on the number of non-conformances found at each severity level for each configuration item andiron-conformances should be tracked to closure. Other useful information to have is the status of the non-conformances (i.e., found, fix identified, fix implemented, fix tested/closed).
SA Tasking for SWE-201 is related to this requirement and additional information that needs to be captured on the non-conformances is below:
- Software problem/Software defects/software change report status:
- total number,
- number closed,
- the number opened in the current reporting period,