1. Introduction to Auto-Generated Code

Model-based software development uses a model as the centerpiece of the development process. Software engineers create a model of the system behavior that can then be translated into different languages such as C or Ada by the appropriate code generator. The model is continually refined throughout the development process and can even be executable. For maximum benefit, the generated code should not be modified by software engineers; when changes are desired, the model is revised, and code is generated from the revised model.

Both the modeling tool and the code generation tool are part of a toolset that makes up the model-based development system, referred to in this guidance as MBDS. The code resulting from MBDS is referred to as auto-generated code.

Efficiency is a key benefit of using MBDS, but there are many considerations to be reviewed when deciding whether MBDS is the right choice for a project.

The need for safe, reliable and secure execution of software is just as applicable to auto-generated software as to software developed by any other means.

2. Guidance for Assuring Auto-generated Code

Assurance for auto-generated code covers two main areas:

1. 1. assurance of the decision of whether to use auto-generation, and
   2. when it is used, the assurance practices to follow.

This NASA Software Engineering Handbook Topic addresses area (a) in its Sections 2.1 (Considerations for Choosing Auto-Generated Code -Usage and Needs) and 2.2 (Considerations for Choosing Auto-Generated Code - Pros and Cons), and area (b) in its sections 3.0 (Software Assurance Practices for Auto-Generated Code). Software assurance personnel should become familiar with the software recommendations on that topic.

Past projects have differed considerably in their approaches to, and use(s) of, auto-generation. Several relatively recent surveys: Torchiano et al, 2013

2.1 Considerations for Choosing Auto-Generated Code-Usage and Needs

When deciding to use auto-generation it is also important to determine whether the generated code will be used as-is or require further (manual) modification. Some of the many factors to be considered, starting with this applicability and purposes are listed below:

• Applicability: the applicability of the requirements is determined by the software classification and the software safety criticality and is specified by the requirements mapping table in Appendix A of NPR 7150.2

  Swerefn
  refnum 083

• Purpose: the auto-generated code can:
  • become (part of) operational code (for flight and/or ground facilities),
  • be used to test of operational code, or
  • be used for rapid prototyping or simulation.
• These different uses lead to different needs and constraints. For example, maintenance of operational code will extend through the lifetime of the mission, whereas maintenance of testing and prototyping code may no longer be needed once development has completed. Flight code’s runtime performance on the flight computer(s) will typically be important, whereas simulation code may be provided with additional computational resources as needed. Note that as per NASA-STD-8739.8A

  Swerefn
  refnum 278

  (Software Assurance and Software Safety Standard), software that provides full or partial verification or validation of safety-critical systems, including hardware or software subsystems (e.g., this can include models and simulations) can also be safety-critical. Also, be aware of NASA-STD-7009

  Swerefn
  refnum 272

  (Standard for Models and Simulations) when considering the use of modeling and simulation.
• Generator technology: the code generator may consist of, or combine, a commercial product, open-source technology, an institutional asset reused on multiple projects, or “homebrew” project-specific developments. Different implications stem from these, e.g., COTS-like considerations for commercial products, sustainment and maintenance considerations for “homebrew” technology.
• Input: the inputs to the code generator can be of different formats/representations (e.g., different modeling languages such as Stateflow, Simulink, or AADL, XML specifications, database schema, etc.). The choice of inputs to code generator could impact the flexibility (e.g., a proprietary format may restrict the ability for the input to be reused as input to other generator tools) and complexity of the resulting inputs. The suitability of the modeling representation with respect to domain and behavior modeled can affect the effort needed to create the inputs/models. Additionally, inputs to the code generator may be reused from other projects; and therefore, additional assurance activities to address concerns/risks similar to code reuse are needed.
• Interface/Integration: the generated code may be of fully executable code or only partially executable (e.g., form a skeleton of the software or implement a database). Additionally, generated code may be of a stand-alone code or integrated with legacy code.
• Post-generation-editing: a major distinction is between auto-generated code that is to be used “as-is” (i.e., once generated, it will not be further edited) vs, the anticipated need for further editing after generation.
• V&V: the amenability of generated code to V&V practices may vary (e.g., as mentioned earlier, auto-generated code may be more voluminous and/or harder to understand than would be the equivalent hand-developed code); similarly, the amenability of the auto generation’s inputs to V&V practices may also vary (e.g., offering opportunities for formal analysis)

2.2 Considerations for Choosing Auto-Generated Code-Pros and Cons

This section lists some of the advantages and disadvantages of using auto-generated code which also needs to be considered when deciding whether to use auto-generated code.

When trying to determine whether to use MBDS and auto-generated code, it is useful to consider the characteristics usually thought of as advantages (or pros) as well as those characteristics which have been identified as disadvantages (or cons.) The most commonly sited pros are productivity, simplification, portability, and consistency.1

Productivity: The developer provides inputs to the code generator, and it generates the code, based on the model and the inputs, usually producing code at a much faster rate than can usually be done with hand-coding. In addition, once the code generator is set up, it can be used many times.

Simplification: Code is generated from an abstract description or model. Typically, it is easier to analyze the abstract description or model then it is to analyze the code. Changes can be made in the models and then the code generator can be rerun to produce code with the correct changes.

Portability: Once there is a process for generating code in one language, the code can be changed to another language by changing the code generator being used.

Consistency: When using a code generator, the generated code always follows the principles, types, and rules set up for the generator. Thus, the quality of the code is consistent.

The most common disadvantages or cons of auto-generated code are maintenance and complexity.

Maintenance: Once the code has been generated, the users are dependent on the code generator and it will need to be updated periodically to stay up to date. There are often issues with finding/keeping a developer who can maintain the code generators.

Complexity: The auto-generated code tends to be more complex than the hand-generated code. It is also less optimized than the hand-generated code. This bulk and complexity may make it more difficult to analyze and correct the code if the code generated is not producing the expected results.

2.3 Assurance of the decision of whether to use auto-generation

The role of Software Assurance should be to check that a project’s decision to use auto-generation is well-founded, i.e., the project has identified, understood and appropriately taken into account the full range of relevant considerations. This should cover how the project plans to address any additional challenges that auto-generation might pose, as well as how the project may be able to take advantage of additional opportunities. In the course of this, Software Assurance may be of help by offering advice and recommendations as appropriate.

A good starting point and a rich source of information on this topic are the SEI’s Technical Report “Model-Driven Engineering: Automatic Code Generation and Beyond” Klein et al, 2015

3.0 Software Assurance Practices for Auto-Generated Code

3.1 Assurance practices to follow when auto-generation is used

Assurance practices specifically required in the NASA-STD-8739.8

Tasking for SWE-070 calls for confirming the software models has been validated and accredited. Software engineering is responsible for the actual validation and accreditation of the models, and assurance just needs to confirm that this has been done. In the case of the models associated with the auto-generation of code, it is important to make sure that the model is correct as well as the code-generator and that the inputs being used produce the expected results when the code generation tools re being used.

Tasking for SWE-081 lists models as an example item is included in those items that are identified to be configuration managed. The models associated with auto-generated code should definitely be configuration managed. SWE-146, part g also calls for the inputs to the auto-generation tool, the outputs of the auto-generation tool and any modifications to the outputs of the auto-generation tool to be configuration managed. Software assurance will confirm that all of these are identified as configuration items and are being configuration managed.

Tasking for SWE-146 calls for software assurance to assess the software plans, processes, and procedures for the approaches that satisfy the bulleted list of conditions in SWE-146. Software assurance will review the software plans, processes and procedures and 1) Confirm that approaches have been developed for the use of auto-generated code and that they include the process of validating and accrediting the auto-generation tools. The process for validating these tools should be consistent with the standard being used to validate and accredit any tools for hand-generated code. 2) Monitor the planned use of auto-generated code versus the actual use of it. 3) Confirm that the limits of the allowable scope of use for the auto-generated code have been described 4) Confirm the following items are identified as configuration items and are being kept under configuration management: models and code generators for auto-generated code as well as the inputs to the tools, the outputs of the tools and any modifications to the output of the tools.5) Confirm that the processes and procedures include those needed for making manual changes to the auto-generated source code.

Tasking for SWE-147 calls for a confirmation that the auto-generated tools have been considered for reusability.

Tasking for SWE-206 calls for software assurance to confirm that NASA engineering, project, software assurance, and IV&V have electronic access to the auto-generation tools, including the models, simulations, and associated data that is used as inputs for auto-generated code.

In some cases, the generated code is more voluminous and/or harder to understand than a hand-developed code. Hence, there is the need to complement the code review activity by also scrutinizing its inputs (models or specifications) and the code generator tool itself. This is aided by the fact that the input is simultaneously rigorous (i.e., has a formal semantics) yet concise and clear (e.g., in a modeling notation well suited to expressing the software’s requirements). These considerations motivate the requirement to configuration to manage the inputs and outputs of the auto-generation of software (SWE-146, part g ) and the requirement to describe the limits and allowable scope of the auto-generated code (SWE-146, part c). Auto-generation tools can assist a project in developing a large amount of code in a relatively short period of time, but it is critical to know exactly how to specify the inputs to generate the desired resulting code.

Requirements pertaining to the code itself (e.g., analysis and testing) apply to auto-generated code just as they would for code created by any other means. The testing of auto-generated code is much like the testing of COTS, and it should be tested to the same degree as any hand-developed code.

More information on guidance for these specific tasking requirements can be found in the Guidance section for each of the associated SWE requirements.

In addition to these assurance tasks, software assurance should consider the following section that lists lessons learned associated with the use and assurance of auto-generated code.

3.2 Assurance Practices and Considerations obtained from Lessons Learned on NASA Projects

This section presents a series of lessons learned from several NASA flight projects employing auto-generation. These cover many topics that assurance should be aware of.

Areas of decreased concern (and hence potentially in less need of assurance):

• There may be less need for human scrutiny of the auto-generated code as compared to manually crafted code, with attention instead shifted to the inputs to auto-generation. e.g., formal inspections of auto-generated code may no longer be conducted, with reviews instead focused on the models' input to auto-generation.
  
  
• Similarly, since the structure and the formatting of the auto-generated code are somewhat dependent on the generator tool, the project may consider relaxing or adjusting the required coding standard. However, the focus may shift to establishing standards for the inputs (model) instead. It is still important to ensure that the generated code is free of coding vulnerabilities (including security-related vulnerabilities), such as buffer overflow, etc., by running the code against static analysis tools. This concern may be less important when a more mature or certified code generator is used.

Areas of new or increased concern (and hence potentially in need of additional assurance):

• The learning curve for engineers not already familiar with the auto-code tooling (e.g., the modeling environment) should not be underestimated.
  
  
• Auto-code generation shifts the development effort from coding implementation to generating inputs to a code generator (modeling). While the average effort needed for coding implementation is generally understood (via extensive metrics collection and industry benchmark available), modeling effort is less well understood. This could pose a risk when evaluating the feasibility of the evaluation plan. Greater monitoring and comparison between planned vs actual may be needed.
  
  
• With increased emphasis on the inputs to auto-generation, standards on the form of those inputs may need to be developed and applied; such standards should be appropriate to the project and form of auto-generation.
  
  
• A common recommendation is that the auto-generated code not be manually edited after generation. Instead, if there is a need to change the code, the recommended practice is to change the model and re-generate the code. However, if manual editing of auto-generated code is performed, it is best to treat that code in the same way as manually created code.
  
  
• Reverse engineering (also referred to as “round-trip engineering”), in which manual edits to the auto-generated code are used to automatically update the inputs to auto-generation (thus keeping the inputs and outputs in sync), is sometimes claimed as a capability (e.g., Hinchey at al

  Swerefn
  refnum 653

  ). In general, this is difficult to do, so this capability should be assured if it is to be relied upon.
  
  
• Similar to the above concern, establishing traceability between the generated code and the inputs (models) can be useful for debugging and maintenance (especially if auto-generated code is to be manually edited after generation).
  
  
• Ideally, auto-generated code not subject to further editing would not be checked into a Configuration Management repository – just the inputs from which to generate the code would be checked in. However, experience has shown that code generation can be time-consuming. Checking in auto-generated code is a means to save on build time – but in such a case be especially careful with its configuration management!
  
  
• Configuration management of the graphical models that often serve as input to auto-generation poses some increased challenges, as outlined in section 5.3.2.
  
  
• Compatibility between auto-generated code and other forms of code should be considered early, given that the auto-generation process may offer less flexibility in adjusting the form of the code it generates.

3.3 Best Practices

Orion Modeling Guideline – designed for modeling using Simulink, the joint Orion NASA/contractor team developed this from the starting point of the Mathworks Automotive Advisory Board (MAAB) guidelines document

Below is an illustration of modeling best practices, abstracted from the aforementioned Orion Modeling Guideline.

3.4 Assurance Considerations for Small Projects

Small projects may not need to be as concerned with elaborate configuration management practices of the input models to auto-generation, especially if they are conducted by a small, cohesive co-located team. If projects are short-term in nature, they may be relatively invulnerable to revisions of third-party auto-generation tools, able to stick with the same version for the duration of the effort. An informal approach to determining and promulgating best practices among the development team may suffice (as compared to developing rigorous standards). The models themselves may be relatively simple (e.g., have fewer instruments and less redundancy to control).

4. Lessons Learned for Consideration

There are many other lessons learned from NASA projects which software assurance personnel should be aware of and make use of when performing software assurance on projects that are making use of auto-generated code.

4.1 Auto-generation to achieve productivity and reviewability

JPL’s 1997-1998 auto-generation of fault protection code for the DS1 spacecraft, described in Rouquette et al

• The clear motivation for use of auto-generation. Schedule pressure (short timeframe and “too few software engineers available”) would have made conventional software development “challenging.” Code generation was chosen in part because it “allowed us to make an important separation of concerns between what system-level FP [Fault Protection] should do (design) and how it should do it (implementation).”It was not just the productivity gain from avoiding a manual coding step (a benefit widely claimed for auto-generation). It also made the functionality being implemented more reviewable by virtue of the form of the input to auto-generation: “We needed a design notation sufficiently clear to allow several people to follow an analysis discourse and sufficiently compact to facilitate such reviews”. And later“…reviewable functionality was the most important criteria to enable the kind of high-level peer design reviews necessary to meet our schedule.”
  
  
• Feasibility of the approach. One set of concerns was whether the auto-generation approach they had in mind (using an existing commercial code-generation tool) would be feasible. Included in this were questions of whether the input format would be adequate to represent the richness of behaviors, whether the generator could be trusted to implement the desired logic, and whether much of the team’s time would be spent debugging the code generator. Their solution was to “rapidly prototype the most complex behavior required … and exhaustively test its generated software.”
  
  
• Code generation issues / Avoidance of post-generation manual editing. The generated code must be compatible with the project, both with the project’s processes and with other code in the project. In the case of incompatibility, the code generator must be sufficiently customizable and/or the project must adapt. The DS1 fault protection effort “… chose a code-generation approach that espoused a zero customization policy towards the generated software.” As they put it, “Custom code becomes one more source of information that needed to be managed.” The customization they needed was attained by using the “template” capability provided by the commercial code generator they employed, and they also modified the (commercial) code generator itself(!).

4.2 Development of an in-house auto-coder

The success of DS1’s auto-generation of fault protection code led to its use in a similar effort, to develop the fault protection code for the Deep Impact (DI) mission. However, by the time of DI the commercial auto-coder had been updated (involving a changed file format), and DI had different requirements for auto-coder output (e.g., required C++ instead of C code). To address this, the DI team wrote a new postprocessor to convert the auto-generated code into the flight C++ code. Following the DS1 and DI experiences, JPL decided to develop an in-house auto-coder. While this had not been practical for the schedule-challenged DS1 mission, time and resources later permitted its consideration.

• The rationale for developing an in-house auto-coder. As stated in Wagstaff et al

  Swerefn
  refnum 651

  , these were to provide:
  • “a non-proprietary file format in which the state charts [the input to the code generation] are stored”,
  • “precise control over which programming constructs are used, to ensure adherence with flight software requirements”, and
  • “the ability to plug in different front-end state chart drawing tools and different back-end output modules”. The “back-end output modules” were to allow not only code (either C or C++) to be generated, but also, from the same inputs, simulator output for execution in simulation via an interactive interface, and Promela output for feeding into the model-checker SPIN.
    
    
• Benefits to quality assurance. The following beneficial influences on quality assurance are reported in Wagstaff et al

  Swerefn
  refnum 651

  :
  • “Code reviews can be made more efficient when an accompanying state chart is available, especially one that can be examined dynamically (in simulation). In addition, after thorough review and validation of the automatic code generation tool, review efforts for an individual project can be focused on the designs (state charts) rather than the implementation.” and along these lines:
  • “During a Preliminary Design Review … [the] simulator was used to catch, and later resolve, divergent design interpretations by designers and implementers”
    
    
• V&V of the auto-coder itself. The JPL developed auto-coder software is classified as NASA Class B (mission-critical) code. It seems that few commercial auto-coders have been rigorously qualified, a rare but notable exception being the SCADE Suite ® described as “qualified as a development tool for DO-178B software up to Level A, etc. (see https://www.ansys.com/products/embedded-software/ansys-scade-suite for the full list of such).

Model-checking using auto-generated output specifically for this purpose is alluring as a means to validate a model against correctness properties (expressed as assertions to the model checker). This seems to have been successful as the means to guarantee some straightforward properties (e.g., to show all states in the state chart are reachable), but not necessarily all the properties one might wish to check, due to scalability issues of model-checking. The interested reader is referred to Oh et al, 2008

5. Resources

5.1 References

5.2 Tools

5.3 Auto-Generated Code Specific Tools

As mentioned earlier, a good starting point for considerations when choosing auto-coding is Klein et al, 2015

The following two tools and documentation are available for download from the Software Assurance Research Program sub-community on the NASA Engineering Network. These tools are available only to NASA Civil Servants and Contractors.

Model-Based Software Development (MBSD) Checklist Tool – A dynamic checklist of context-driven software assurance concerns and practices:

1. Documentation
2. Checklist tool (Excel .xlsm file, including macros)

Complexity Metrics for Simulink Models – A script to calculate several metrics of Simulink models

1. Documentation
2. Script (.zip file)