1. Requirements

1.1 Notes

Although no maximum cyclomatic complexity score is required for non-safety-critical software, all software projects should regularly collect and maintain complexity metrics and use them to manage risk, either when high-complexity code must be modified, or proactively to improve the overall quality and maintenance of the code base. For safety-critical software, the analysis should take into account the requirements for cyclomatic complexity and code coverage as defined in 3.7.5 and 3.7.4 respectively.

1.2 History

1.3 Applicability Across Classes

2. Rationale

The static analysis requirement for NASA software projects increases the quality and safety of code developed for NASA Missions. Using static analysis helps to ensure that code meets the coding standards/criteria established by the project team (SWE-061 - Coding Standards) and common coding errors are eliminated before system integration and test (SWE-066 - Perform Testing). Studies show that the cost of catching errors dramatically increases from one phase of development to the next by roughly a factor of 10 . Eliminating errors during implementation results in cost and time savings during integration and testing, which is a particularly important cost-saving factor for projects using high-fidelity testbeds.

3. Guidance

3.1 Modern Static Analysis Tools

Static analysis tools are any tools that are used to analyze code that is not running (ie “static”). Modern static code analysis tools can identify a variety of issues and problems, including but not limited to dead code, non-compliances with coding standards, security vulnerabilities, race conditions, memory leaks, and redundant code. Modern static code analysis tools can identify a variety of issues and problems, including but not limited to dead code, non-compliances with coding standards, security vulnerabilities, race conditions, memory leaks, and redundant code. Typically, static analysis tools are used to help verify adherence with coding methods, standards, and/or criteria. While false positives are an acknowledged shortcoming of static analysis tools, users can calibrate, tune, and filter results to make effective use of these tools. Software peer reviews/inspections of code items can include reviewing the results from static code analysis tools.

See also Topic 8.19 - Dead / Dormant Code and Safety-Critical Software

Static analysis tools may not be readily available for some platforms or computing languages. If static analysis tools are determined to be unavailable, the project can document the alternate manual methods and procedures to be used to verify and validate the software code. These manual methods and procedures will be addressed or referenced in the project's compliance matrix against this requirement.

For critical code, it is essential to use sound and complete static analyzers. Sound and complete analyzers guarantee that all errors are flagged and that no false negatives (i.e., an erroneous operation is classified as safe) are generated. Such commercial analyzers are expensive but necessary for critical applications. Note that sound and complete static analyzers are now available free of charge for C and C++ software systems (see the Tools Table provided under the Resources tab).

See also SWE-040 - Access to Software Products for details on getting access to code for conducting an analysis. 

See also SWE-190 - Verify Code Coverage, 

For Class A software take care to analyze all software affecting safety-critical software  and hazardous functionality including: HR-33 - Inadvertent Operator Action

3.2 Cost Of Tools

The cost of sound and complete commercial tools can be prohibitive for projects working on non-critical software systems. However, there are other static analyzers available free of charge, which can identify many common errors or deviations from standards or good coding practices. It is a best practice to use some type of static analysis in all software development efforts but is required for mission-critical software.

3.3 Characteristics Of Modern Tools

Introducing the routine use of static analyzers in an existing development process can be a tricky proposition. Even if the software team is convinced of the benefits such tools can bring, projects need to be careful and make the introduction of static analysis as unobtrusive as possible. Fortunately, current generation static analyzers come with interfaces similar to compilers, such as GNU (Unix-like computer operating system developed by the GNU Project) Compiler Collection (GCC), and/or Integrated Development Environments (IDEs), such as Eclipse, which helps in gaining acceptance. However, the user interface is not the only factor to consider. Here is a non-exhaustive list of potential challenges:

3.3.1 The static analyzer may not accept your code.

It is not uncommon that the first problem (after installation) with a static analyzer is its inability to process code.

Today, most commercial static analyzers rely on the same front-end code processor (e.g., the Edison Design Group (EDG) front-end for C and C++), which means that they in principle can process the same code. However, the algorithms used for the analysis might not handle code constructs in similar manners, and therefore, there are differences amongst tools. This problem is even more prevalent with tools that are free of charge because they may rely on different parsers, so commercial tools are recommended.

Guidance: For existing projects interested in evaluating commercial static analyzers, consider the following pointers:

• • Start with a static analyzer that comes with a free evaluation period and evaluate it on your most “tricky” code, i.e., with all the language constructs and libraries that the team commonly uses.
  • Choose a commercial analyzer based on your evaluation, not on a vendor demonstration; vendors will only show what works, not the potential challenges.
  • Choose a tool that comes with extensive technical support. Your application might have quirks that do not seem like quirks to your project but might be to others. So, readily available support can be a key factor in effectively using a static analyzer. For this reason, commercial tools are recommended. Free static analyzers will not generally provide the support the software team may need.
  • Choose a tool that relies on commonly used parsers to mitigate future issues. For example, for C and C++, it is best to choose a tool that relies on GCC for processing code. In general, GCC can process all C and C++ programs and also offers options to cover old constructs or customized extensions. 

3.3.2 The static analyzer may take an extensive amount of time to finish the analysis.

In cases where the analysis is in-depth and code is “complex”, a static analyzer can take an inordinate amount of time to come back with results. Using a static analyzer includes making configuration decisions regarding precision, scalability, and language construct coverage. The most precise analyzers use sophisticated algorithms and usually have large memory requirements. These factors may result in slow analysis times and in running out of memory (which causes swapping with “slow” storage mechanisms, e.g., hard disks).

Guidance: Select and configure a static analyzer to operate within a diurnal development cycle and consider two analysis steps. As a rule of thumb, don’t set up a static analysis session to take more than eight hours. This will ensure that you can run overnight and offer new results every morning.  Note that the most precise static analyzers come with different levels of analysis, which can be configured to fit time requirements and later to run a precise analysis of the project at select points in the development process.  Fast but imprecise analyzers can be added to produce quick results but will produce more warnings. Care must be taken to filter results, remove irrelevant findings, and also configure the tool without suppressing useful, if noisy, code checks. 

3.3.3 The static analyzer comes back with too many warnings.

Fortunately, there are now many mechanisms (filtering, analysis tuning, increased precisions) to deal with this problem. This challenge also stems from the need to trade precision for scalability. For example, you may configure an analyzer to be complete, which means that it needs to flag every possible error. In doing so, it relies on approximations (called abstractions in the literature), which lead the analysis to consider execution paths that may not be realizable.  Therefore the analyzer may not distinguish between a real error and a possible error (warnings). The question is: “What can be done when faced with an overwhelming number of warnings?”

Guidance: First, make sure you are familiar with the static analyzer. Static analyzers that look for errors fall into two categories: debuggers and certifiers. Debuggers try to find bugs quickly, and in doing so, they cut many corners; thus they usually generate many warnings. Certifiers are slower but they try to characterize every operation as safe, unsafe, potentially unsafe, or dead. Depending on your need, pick one or the other, or use one or the other at the proper place in your process. Note that certifiers are recommended for safety-critical code.

Second, make sure that your static analyzer has a filtering capability. Most come with a means of filtering by types of errors. However, if possible, pick a tool that can rank warnings so that you can focus on the most likely bugs. Some tools will do that automatically and present only a percentage of all the warnings. Lastly, it may be necessary to write a customized script to filter results.

3.4 Static Analysis And Manual Code Reviews

Checking coding standards/criteria through the use of a static analysis tool complements manual code reviews (SWE-087 - Software Peer Reviews and Inspections for Requirements, Plans, Design, Code, and Test Procedures), and results in safer software systems. Coding standards/criteria (see also SWE-061 - Coding Standards) ensure code consistency across a project and facilitate not only reviews but also software system integration. Software components that do not comply with coding standards can result in incorrect assumptions about the component, which can result in errors at later development stages. Debugging can also be hampered by deviations from coding standards/criteria since they can result in misleading assumptions. Enforcement of coding standards yields benefits that can continue into the later software life cycle phases, reducing software maintenance complexity, and fostering code reuse.

See also Topic 8.26 - Static Analysis. 

3.5 Availability Of Tools

The static analysis applies to a wide range of software systems, with exceptions when tools are not available for a particular platform or programming language. Note that there are many available tools for widely used languages such as Ada, C, C++, and Java (see the Tools Table referenced under the Resources tab).

The NASA IV&V Program maintains access to an extensive set of static code analyzers to support the evaluation of a diverse set of coding languages including C, C++, Ada, Java, and Fortran.  IV&V has established a robust infrastructure to support the independent, meaning outside of the developers' software development environment (SDE), execution of these tools as well as a mature capability for the prioritization and evaluation of the analyzer results.  The IV&V Program regularly utilizes these analyzers in the evaluation of the Agency’s safety and mission-critical software and can provide fee-for-service support for projects not currently receiving IV&V from the NASA IV&V Program.  For more information please contact the IV&V Program business representatives via the IV&V Program website .

3.6 Additional Guidance

Additional guidance related to this requirement may be found in the following materials in this Handbook:

3.7 Center Process Asset Libraries

See the following link(s) in SPAN for process assets from contributing Centers (NASA Only). 

4. Small Projects

Small projects can benefit from inserting static analysis in their development process. As mentioned above, there are many static analyzers available free of charge. So, using these does not impose an additional cost on a small project with constrained resources. However, it might have an impact on development time, since it takes time to get used to a new tool and free tools tend to generate a lot of false positives.

The time spent getting used to a tool is negligible. Most tools use the same interfaces as compilers and are not difficult to use. There might be differences due to the language parser used by the tool. To ensure smooth integration of the tool in the development process, pick a tool that relies on parsers from common compilers (e.g., GCC). This helps to ensure that the tool can be easily integrated into existing make-files or other compiling mechanisms.

The generation of many false positives can be a problem since it might overwhelm the users. In some sense, it is similar to the problem of choosing an adequate warning level for a compiler: generating too many warnings causes them to be ignored. Choose a static analyzer that has the capability of filtering results or that can be adjusted to generate fewer warnings. In general, it is a good habit to start using a static analyzer at the level that generates fewer warnings and to slowly increase the levels until too many warnings are generated.

5. Resources

5.1 References

5.2 Tools

6. Lessons Learned

6.1 NASA Lessons Learned

A documented lesson from the NASA Lessons Learned database notes the following:

• Independent Verification and Validation of Embedded Software. Lesson Number: 0723: This lesson learned is based on Reliability Practice No. PD-ED-1228; from NASA Technical Memorandum 4322A, NASA Reliability Preferred Practices for Design and Test.
  
  "Implementation Analysis Phase: During this IV&V phase, two parallel activities are performed: (1) coding analysis and (2) testing. Coding analysis includes version comparison, textual and syntactical analysis, standards auditing, equation reconstruction, data structure analysis, flowcharting, logic reconstruction, manual code inspection, traceability analysis, interface analysis, and database analysis. Software tools are employed to automate many of these program analysis techniques. They are used to help identify actual or potential errors in the developed code, and to reformat and consolidate information to facilitate manual analysis, software tools present a reliable, cost-effective means to supplement manual program analysis techniques. To maximize the visibility of software development quality, coding analysis is performed in parallel with code development. Coding analysis is achieved by analyzing the incremental code deliveries and modifications introduced in the updated program versions."
• Static Software Analysis of the NASA Autonomous Flight Termination Software Lesson Number 24503  : 
  • Software development teams should specify specific coding standards to be applied to the software development for both ground utilities and flight software.
  • Software development teams should configure the static analysis tool to best match the specified coding standard. Matching the coding standard and the static analysis tool capability leverages the tool as a verification asset.
  • Software development teams should define how the reported issues are to be reviewed, and what defines an issue that should be corrected.

6.2 Other Lessons Learned

The Department of Defense publication Crosstalk Magazine contains the following lessons learned related to static analysis tools:

1. Challenges in Deploying Static Analysis Tools. For higher quality software and competitive products, many projects are feverishly deploying static analysis tools. Unfortunately, it turns out that many of the deployments are failures. Some have discontinued static analysis tools altogether. Some continue to use them, but find that the results are not as effective as they hoped.
   Many challenges are facing static analysis tool deployments. Although static analysis tools have some weaknesses, the main challenge stems from people. Whether the tool deployment succeeds or fails depends on the people behind it. What are the challenges facing static analysis tool deployments and how can those challenges be overcome? This paper tries to answer that question based on our deployment of the tools, consultancies with other organizations, and others' experiences.
2. Software Static Code Analysis Lessons Learned. The United Kingdom Ministry of Defense has been at the forefront of the use of software static code analysis methodologies, including some of the tools and their application. This article ... discusses what is meant by static analysis, reviews some of the tools, and considers some of the lessons learned from the practical application of software static code analysis when used to evaluate military avionics software .

7. Software Assurance

7.1 Tasking for Software Assurance

7.2 Software Assurance Products

• Source Code Analysis
• Static Code analysis results are captured as corrective actions with appropriate metadata.

  
  

  Static analysis results (from engineering, software assurance, or IV&V)

7.3 Metrics

• Static Code Analysis Metrics:
  
  • Document the Static Code Analysis tools used with associated Non-Conformances
  • # of total errors and warnings identified by the tool
  • # of errors and warnings evaluated vs. # of total errors and warnings identified by the tool

  • # of static code “positives” over time (Open, Closed, Severity)

  • Total # of static code analysis "positives" vs.  # of "positives" resolved. Trend over time.
  • # of static code errors and warnings identified as “positives” vs. # of total errors and warnings identified by the tool
  • The trend of # of total errors and warnings identified per SCA Tool, Language, and SLOC size
  • # of Non-Conformances raised by SA vs. total # of raised Non-Conformances
  • # of static code errors and warnings resolved by Severity vs. # of static code errors and warnings identified by Severity by the tool

• Cybersecurity specific metrics
  • # of Cybersecurity vulnerabilities and weaknesses identified
  • # of Cybersecurity vulnerabilities and weaknesses (Open, Closed, Severity)
  • Trending of Open vs. Closed over time
  • Trends of Cybersecurity Non-Conformances over time
  • # and type of vulnerabilities and weaknesses identified by the project
  • # of Cybersecurity vulnerabilities and weaknesses identified by the tool
  • # of Non-Conformances identified in Cybersecurity coding standard compliance (Open, Closed)
  • # of Cybersecurity vulnerabilities and weaknesses identified vs. # resolved during Implementation
  • # of Cybersecurity vulnerabilities and weaknesses identified by life cycle phase
• # of safety-related Non-Conformances identified by life cycle phase over time

See also Topic 8.18 - SA Suggested Metrics. 

7.4 Guidance

1. Analyze the engineering data or perform independent static code analysis to check for code detects defects, software quality objectives, code coverage objectives (if required), software complexity values (cyclomatic and dependencies between modules), and software security objectives.
2. Confirm the static analysis tool(s) are used with checkers to identify security and coding errors and defects.
3. Assess that the project addresses the results from the static analysis tools used by software assurance, software safety, engineering, or the project.
4. Confirm that the software code has been scanned for security defects and confirm the result.
5. Per SWE-219 - Code Coverage for Safety Critical Software for safety-critical software, verify code coverage and approved waivers.
6. Per SWE-220 - Cyclomatic Complexity for Safety-Critical Software for safety-critical software, verify cyclomatic complexity and approved waivers.
7. Confirm that Software Quality Objectives (architecture, coding standards, maintainability, fault tolerance, security, testability) or software quality threshold levels are defined and set for static code analysis defects, checks, or software security objectives.

Software assurance needs to confirm that static analysis tools are being used and include, at a minimum, the capabilities to identify security issues as well as coding errors and defects. If the code is safety-critical, it is important to choose a static analyzer that works well for verifying whether operations are safe, unsafe, potentially unsafe, or dead. These are typically called “certifiers” and usually run slower but produce better results for safety-critical software. “Certifiers" are highly recommended for checking safety-critical code.

There are many different kinds of static analyzers with different capabilities. Some may cover one type of problem better than other types, for example, some analyzers do not look for security issues, while others make that a primary focus. Also, not all static code analyzers run on all code languages. Many are language-specific. Thus, it is important to choose a code analyzer that is best for the type of code being tested. It is recommended that more than one code analyzer or certifier be run on the code, particularly for more critical projects. The guidance for software in this requirement has a lot of useful information on code analyzers and can be used to help determine if the project has chosen code analyzers that are appropriate for the project. 

The cost of static code analyzers varies considerably, from free to quite costly, so that is another thing to consider when choosing your analyzers/certifiers. The IV&V Facility has several code analyzers which they can use on a fee-for-service basis to help the projects not receiving IV&V.

In the case where no static code analyzers are available to run on the project software, alternate methods of finding potential errors can be used, but they have to be listed and approved in the project compliance matrix.

After the code analyzers have been run, software assurance will assess whether the project is addressing the issues and defects identified by the code analyzers. All issues and defects found should be examined for false positives, and then the remaining defects and issues should be prioritized in order of severity to the system. All issues and defects relating to safety or security problems need to be fixed/resolved. The project should also fix/resolve all high and medium problems and as many of the low or cosmetic issues as possible. Software assurance can help the project determine the risk of not fixing certain types of errors.

See also SWE-157 - Protect Against Unauthorized Access, SWE-185 - Secure Coding Standards Verification, 

7.5 Additional Guidance

Additional guidance related to this requirement may be found in the following materials in this Handbook: