1. Requirements

3.11.7 The project manager shall test the software and record test results for the required software cybersecurity mitigation implementations identified from the security vulnerabilities and security weaknesses analysis. 

1.1 Notes

Include assessments for security vulnerabilities during Peer Review/Inspections of software requirements and design. Utilize automated security static analysis as well as coding standard static analyses of software code to find potential security vulnerabilities.

1.2 History

1.3 Applicability Across Classes

Key:    - Applicable | - Not Applicable

2. Rationale

Mitigations identified to address security risks need to be confirmed as appropriate and adequate.  Software verification and validation (V&V) processes are used to determine whether the software security risk mitigations, as planned and implemented, satisfy their intended use to address security risks in space flight software.  The results of V&V activities serve as a documented assurance that the correct mitigations were identified, implemented, and will reduce to an acceptable level or eliminate known software security risks.

3. Guidance

Project managers ensure that software security risk mitigations called out in the Project Protection Plan for space flight software are verified and validated as part of the project’s development activities. The verification and validation (V&V) of these mitigations are included in the project’s planned V&V activities, reflected in the project’s V&V plans, and carried out throughout the project life cycle.

When planning V&V for required software security risk mitigations, the following are useful source documents within the project and from the Information System Security Officer (ISSO):

• System-level risk assessment report.
• Plan of actions and milestones (POA&M). (POA&M serves as the NASA management tool to address, report, resolve, and remediate security-related weaknesses, both at the information system level and program-level).

Examples of the type of information found in these documents relevant to planning V&V for software security risk mitigations include:

• Vulnerabilities discovered during the security impact analysis or security control monitoring.
• Corrective efforts associated with the mitigation of identified security weaknesses.
• The NASA goal for vulnerability remediation was identified during vulnerability scanning.
  • To have all vulnerabilities categorized as high remediated within five working days of discovery.
  • To have all vulnerabilities categorized as moderate remediated within thirty working days of discovery.
  • To have all vulnerabilities categorized as low remediated within sixty working days of discovery.

Specific to software security risk mitigations, V&V activities include, at a minimum:

• Security vulnerability checks during peer review/inspections of software requirements, design, code (may require updates to existing peer review/inspection checklists).
• Automated security static code analysis.
• Coding standard static analysis.

Additionally, repeating previous assessments for software security vulnerabilities could be used to confirm that the previously identified vulnerabilities and weaknesses have been addressed and no new vulnerabilities introduced or discovered.  To keep costs down and prevent the propagation of risks through the project, mitigations are to be put in place as early as possible and verified throughout the life cycle.

Additional guidance related to software security and verification and validation may be found in the following related requirements in this Handbook:

4. Small Projects

No additional guidance is available for small projects. 

5. Resources

5.1 References

5.2 Tools

6. Lessons Learned

6.1 NASA Lessons Learned

No Lessons Learned have currently been identified for this requirement.

6.2 Other Lessons Learned

No other Lessons Learned have currently been identified for this requirement.

7. Software Assurance

7.1 Tasking for Software Assurance

1. Confirm that testing is complete for the cybersecurity mitigation.
2. Assess the quality of the cybersecurity mitigation implementation testing and the test results.

7.2 Software Assurance Products

• Source Code Analysis
• Verification Activities Analysis
• SA assessment of the quality of cybersecurity mitigation testing.  

  
  

  Objective Evidence

  • Software Test Procedures
  • Software Test Reports
   Definition of objective evidence

  Objective evidence is an unbiased, documented fact showing that an activity was confirmed or performed by the software assurance/safety person(s). The evidence for confirmation of the activity can take any number of different forms, depending on the activity in the task. Examples are:

  • Observations, findings, issues, risks found by the SA/safety person and may be expressed in an audit or checklist record, email, memo or entry into a tracking system (e.g. Risk Log).
  • Meeting minutes with attendance lists or SA meeting notes or assessments of the activities and recorded in the project repository.
  • Status report, email or memo containing statements that confirmation has been performed with date (a checklist of confirmations could be used to record when each confirmation has been done!).
  • Signatures on SA reviewed or witnessed products or activities, or
  • Status report, email or memo containing a short summary of information gained by performing the activity. Some examples of using a “short summary” as objective evidence of a confirmation are:
    • To confirm that: “IV&V Program Execution exists”, the summary might be: IV&V Plan is in draft state. It is expected to be complete by (some date).
    • To confirm that: “Traceability between software requirements and hazards with SW contributions exists”, the summary might be x% of the hazards with software contributions are traced to the requirements.
  • The specific products listed in the Introduction of 8.16 are also objective evidence as well as the examples listed above.

7.3 Metrics

•  # of software work product Non-Conformances identified by life-cycle phase over time
• Total # of tests completed vs. number of test results evaluated and signed off
• # of Safety-Critical tests executed vs. # of Safety-Critical tests witnessed by SA
• # of Cybersecurity Risks with Mitigations vs. # of Cybersecurity Risks identified
• # of Cybersecurity vulnerabilities and weaknesses identified
• # of Cybersecurity vulnerabilities and weaknesses (Open, Closed, Severity)
• Trending of Open vs. Closed over time
• # of Cybersecurity vulnerabilities and weaknesses identified by life-cycle phase
• # of Cybersecurity vulnerabilities and weaknesses identified vs. # resolved during Implementation
• # of tests executed vs. # of tests completed
• # of Non-Conformances identified during each testing phase (Open, Closed, Severity)
• # of Cybersecurity mitigation implementations identified from the security vulnerabilities and security weaknesses
• # of Cybersecurity mitigation implementations identified with associated test procedures vs. # of Cybersecurity mitigation implementations identified
• # of Cybersecurity mitigation tests completed vs. total # of Cybersecurity mitigation tests
• # of Non-Conformances identified during Cybersecurity mitigation testing (Open, Closed, Severity)
• Trends of Cybersecurity Non-Conformances over time
• # of tests completed vs. total # of tests
• # of detailed software requirements tested to date vs. total # of detailed software requirements

7.4 Guidance

The following are steps to be taken to confirm that the quality and completeness of the testing to address cybersecurity mitigation requirements: 

1. Confirm the quality of the testing and the results. A source code quality or weakness analyzer can be used.
2. Analyze static code analysis results and compare to the Software Assurance independent static code analysis and compare baseline results.
3. Perform a peer review with the Software Engineer and go over known vulnerabilities and weaknesses in the software and plans to address them.

Review that the engineering team and the project have addressed any identified cybersecurity vulnerabilities and weaknesses in the software requirements, design, code.  Confirm that the requirements associated with the identified cybersecurity vulnerabilities and weaknesses have been tested or are planned to be tested.  Check to see if the engineering team and the project have run a static analysis tool to assess the cybersecurity vulnerabilities and weaknesses in the source code if so check to see that the findings from the static analysis tool have been addressed by the team.

A method of identifying weaknesses and vulnerabilities is to use the National Vulnerability database from NIST that is the U.S. government repository of standards-based vulnerability data. Software weaknesses can be identified using Common Weakness Enumeration (CWE) - a dictionary created by MITRE.

See the secure coding site ⁶⁶⁴ (NASA access only) for more information.