1. Requirements

2.6.1.1 The project shall require the software supplier(s) to provide insight into software development and test activities; at a minimum the following activities are required:  monitoring integration, review of the verification adequacy, review of trade study data and results, auditing the software development process, participation in software reviews and systems and software technical interchange meetings.

1.1 Notes

NPR 7150.2, NASA Software Engineering Requirements, does not include any notes for this requirement.

1.2 Applicability Across Classes

Classes C through E and Safety Critical are labeled, "P (Center) + SO". This means that this requirement applies to the safety-critical aspects of the software and that an approved Center-defined process which meets a non-empty subset of the full requirement can be used to achieve this requirement

Classes C and D and not Safety Critical and Class G are labeled with "P (Center)." This means that an approved Center-defined process which meets a non-empty subset of the full requirement can be used to achieve this requirement.

2. Rationale

Software supplier activities must be monitored to assure the software work products that are called for in the project and software requirements (see SWE-048) are actually produced. Appropriate use of software project "insight" (surveillance mode requiring the monitoring of customer-identified metrics and contracted milestones) allows NASA to detect problems early and take corrective action if necessary. The insight activities cited in this requirement comprise a minimum set that assures a continual knowledge of the work status achieved by the software developer/supplier when executed over the software product's life cycle.

3. Guidance

SWE-039 specifies minimum NASA "insight" (surveillance mode requiring the monitoring of customer-identified metrics and contracted milestones) activities to be included in the contract with the software supplier. Although the SWE is written to address contract requirements for insight, its contents may also be applied to internal supplier organizations. The minimum activities mentioned in the requirement are to be included as part of the issued Statement of Work (SOW). A listing and description of the final activities agreed to are included in the contract statement of work, for external suppliers, or in a formal agreement between organizations, for internal suppliers. NASA "oversight" (oversight is a surveillance process that implies a more active supervision of a contractor's processes and decision making) activities are not covered in SWE-039. These latter activities are included in the overall project plan, the Software Management Plan (see SWE-102), and the contract SOW. See 7.03 - Acquisition Guidance, and 7.04 - Flowdown of NPR Requirements on Contracts and to Other Centers in Multi-Center Projects for guidance on what to put into contract SOW, advice on how to flow down requirements to suppliers, and, in some instances, suggestions for language for the contract. Consult the Center or program planning documents for help with proposal language. (Proposal information is typically unique and is usually developed by Source Evaluation Boards(SEBs).)

Once a qualified supplier is selected for the software acquisition activity (see 7.03 - Acquisition Guidance ), the software development team still needs to monitor the supplier's activities so the software development work is actually accomplished according to the project and software requirements. Insight into the development of software and the testing of software gives NASA knowledge about the progress being made by the software developer throughout the life cycle.

Insight is a continuum that can range from low intensity, such as reviewing quarterly reports, to high intensity, such as performing surveys and reviews, or witnessing software development or testing activities. Insight begins with attendance at important meetings. It enables the customer to see the work, work products, and supporting processes of the supplier. The level of insight to be exercised on the project is specified in the contract. The use of the insight role to monitor the software developer's progress requires a balance between too much and too little involvement with the software developer.

While insight is defined in one case as a "surveillance mode requiring the monitoring of customer-identified metrics and contracted milestones" (NPR 8735.2, ⁰³⁰), this requirement specifies more. SWE-039 lists five areas that, if properly executed, will provide sufficient knowledge about the state of development of the software. These activities in turn enable the correct evaluation of the metrics and milestones associated with the contract.

The following paragraphs discuss the five areas.

Monitoring Integration

Software integration proceeds from the development of individual components that have coded interfaces up to the time when the team assembles them into integrated systems of coded software and hardware.

Successful software integration depends on whether:

• The hardware is compatible with the software..
• There is agreement about the communications protocols to be used.
• There is understanding about how to transfer information among the information architectures involved.
• The software interrelationships are well defined.
• The information at the interface between systems is in an open format, or mapped to adequate neutral-format standards.
• The purpose of the communication is mutually understood.
• The terminology used is unambiguous in meaning.

Understanding the state of development of these integration characteristics must be a goal of the project's insight activity.

In addition to software development there are also basic COTS and legacy software issues that must be considered during the monitoring of software integration. Evaluation of the following characteristics usually occur during monitoring activities.

• Structure of the architecture and constraints.
• Techniques for mining legacy assets.
• Re-coding and re-hosting.
• Long-term maintenance.
• Translators and emulators.
• Middleware standards and products.
• Management of interfaces.
• Issues resolution.

For a deeper discussion of these characteristics, see http://sunset.usc.edu/gsaw/gsaw98/brkoutsumm2.pdf.

Review of the Verification Adequacy

The requirements described in SWE-028 and SWE-030 cover software verification. The note for SWE-028 describes software verification as the software engineering activity that shows confirmation that software products properly reflect the requirements specified for them. Insight into the adequacy of the verification activities includes examination of the following:

• Objective evidence of the quality of the product from the software developer (e.g., accompanying documentation,  test reports, statistical records, process control).
• Inspections and audits of the software.
• Reviews of the required documentation.
• Reviews of any delegation of verification to the supplier made in the SOW.

When the NASA software team uses periodic reports (e.g., progress reports, or summary and individual test reports) to verify supplier-developed software, the requirements for the data in these reports are specified in the SOW. When software verification will be performed at the supplier's site, the requirements for the method of verification are specified in the SOW. (If software verification will be done by demonstration, consider "COTS Software: Vendor Demonstration Guidelines and Scripts" ¹⁶⁸ for an example of a demonstration script template.)

Review of Trade Study Data and Results

The team exercising the "insight" role uses the contract-specified access to enable the review of all applicable trade studies performed by the software developer or the project.  The performance of adequate trade studies helps assure the correctness of the software development and the software testing choices during the actual software development.

Trade studies compare the relative merits of alternative approaches, and so ensure that the most cost-effective system is developed. They maintain traceability of design decisions back to fundamental requirements. Trade studies do this by comparing alternatives at various levels for the system being developed. They may be applied to concept, design, implementation, verification, support, and other areas. They provide a documented, analytical rationale for choices made in system development.

Trade studies can be used to compare a number of options. For additional background on trade study procedures see MSFC-HDBK-3173, Rev A, [Project Management and Systems Engineering Handbook ³⁶⁰

Auditing the Software Development Process

NASA considers a software audit to be an examination of a work product or set of work products performed by a group independent from the developers to assess compliance with specifications, standards, contractual agreements, or other criteria. This NASA definition is from NASA-STD-8739.8 "Software Assurance Standard," ²⁷⁸ and is based on IEEE 610.12. ²²²

"Software product" mostly, but not exclusively, refers to some kind of technical document. IEEE Std. 1028, IEEE Standard for Software Reviews, ²¹⁹ offers a list of 33 "examples of software products subject to audit,"  including source code and documentary products, such as various sorts of plans, contracts, specifications, designs, procedures, standards, and reports, but also non-documentary products, such as data, test data, and deliverable media.

Software audits are distinct from software peer reviews and software management reviews in that they are conducted by personnel external to, and independent of, the software development organization, and are concerned with compliance of products or processes, rather than with their technical content, technical quality, or managerial implications.

Participation in Software Reviews and Systems and Software Technical Interchange Meetings

The NASA software development team must be afforded the opportunity to attend the regular and periodic meetings that include software development progress reviews, systems engineering meetings regarding software activities, and regular technical interchange meetings. The material reviewed and the ideas for software development that are discussed at these meetings and interchanges often provide a more robust description of the software architecture, the software requirements, the design and discussions of major issues than are typically presented in monthly or quarterly submitted reports.

The meetings, reviews, and technical interchange opportunities must be specified in the statement of work. (See 7.03 - Acquisition Guidance and SWE-048 for advice on oversight activities, specifically what to include for binding reviews, participation on review boards, and how to handle comments and assigned actions.)

If the Defense Contract Management Agency services are available at the software development site, consider using them for basic monitoring and verification services. Arrangements for these services are typically made by the Center's acquisition department.

4. Small Projects

This requirement applies to all projects regardless of size.

5. Resources

5.1 Tools

Tools to aid in compliance with this SWE, if any, may be found in the Tools Library in the NASA Engineering Network (NEN).

NASA users find this in the Tools Library in the Software Processes Across NASA (SPAN) site of the Software Engineering Community in NEN.

The list is informational only and does not represent an “approved tool list”, nor does it represent an endorsement of any particular tool. The purpose is to provide examples of tools being used across the Agency and to help projects and centers decide what tools to consider.

6. Lessons Learned

A documented lesson from the NASA Lessons Learned database notes the following:

Space Shuttle Program/External Tank (ET)/Super Light Weight Tank (SLWT). Lesson Number 1048: A discussion on the shuttle program's certification of its super light weight tank attests to the value of exercising insight and guidance through limited oversight activities ⁵³⁵.