3.12.8 The project manager shall require the software supplier(s) to provide insight into software development and test activities; at a minimum, the software supplier(s) will be required to allow the project manager or designate to:
a. Monitor product integration.
b. Review the verification activities to ensure adequacy.
c. Review trade studies and source data.
d. Audit the software development process.
e. Participate in software reviews and systems and software technical interchange meetings.
NPR 7150.2, NASA Software Engineering Requirements, does not include any notes for this requirement.
1.2 Applicability Across Classes
Classes F and G are labeled with “X (not OTS)”. This means that this requirement does not apply to off-the-shelf software for these classes.
Software supplier activities are monitored to assure the software work products are produced per the project and software requirements. Appropriate use of software project "insight" (surveillance mode requiring monitoring of customer identified and contracted milestones) allows NASA to detect problems early and take corrective action if necessary. The insight activities cited in this requirement comprise a minimum set that assures a continual knowledge of the work status achieved by the software developer/supplier when executed over the software product's life cycle.
SWE-039 specifies minimum NASA insight activities to be included in the contract with the software supplier. Although the SWE is written to address contract requirements for insight, its contents may also be applied to internal supplier organizations. The minimum activities mentioned in the requirement are to be included as part of the issued SOW. A listing and description of the agreed upon final activities are included in the contract statement of work, for external suppliers, or in a formal agreement between organizations, for internal suppliers. NASA oversight (oversight is a surveillance process that implies a more active supervision of a contractor's processes and decision making)) activities are not covered in SWE-039. These latter activities are included in the overall project plan, the Software Management Plan (see SDP-SMP), and the contract SOW. See Topic topics 7.3 - Acquisition Guidance, and Topic 7.4 - Flowdown of NPR Requirements on Contracts and to Other Centers in Multi-Center Projects for guidance on what to put into a contract SOW, advice on how to flow down requirements to suppliers, and, in some instances, suggestions for language for the contract. Consult the Center or program planning documents for help with proposal language. (Proposal information is typically unique and is usually developed by Source Evaluation Boards (SEBs).)
Once a qualified supplier is selected for the software acquisition activity (see Topic 7.3 ), the software development team still needs to monitor the supplier's activities so the software development work is actually accomplished according to the project and software requirements. Insight into the development of software and the testing of software gives NASA knowledge about the progress being made by the software developer throughout the life cycle.
Insight is a continuum that can range from low intensity, such as reviewing quarterly reports, to high intensity, such as performing surveys and reviews, or witnessing software development or testing activities. Insight begins with attendance at important meetings. It enables the customer to see the work, work products, and supporting processes of the supplier. The level of insight to be exercised on the project is specified in the contract. The use of the insight role to monitor the software developer's progress requires a balance between too much and too little involvement with the software developer.
While insight is defined in one case as a "surveillance mode requiring the monitoring of customer-identified metrics and contracted milestones" (NASA-STD 8709.22
), this requirement specifies more. SWE-039 lists five areas that, if properly executed, will provide sufficient knowledge about the state of development of the software. These activities in turn enable the correct evaluation of the metrics and milestones associated with the contract.
The following paragraphs discuss the five areas.
Software integration proceeds from the development of individual components that have coded interfaces up to the time when the team assembles them into integrated systems of coded software and hardware.
Successful software integration depends on whether:
The hardware is compatible with the software.
There is agreement about the communications protocols to be used.
There is understanding about how to transfer information among the information architectures involved.
The software interrelationships are well defined.
The information at the interface between systems is in an open format, or mapped to adequate neutral-format standards.
The purpose of the communication is mutually understood.
The terminology used is unambiguous in meaning.
Understanding the state of development of these integration characteristics must be a goal of the project’s insight activity.
In addition to software development there are also basic COTS and legacy software issues that must be considered during the monitoring of software integration. Evaluation of the following characteristics usually occurs during monitoring activities.
Structure of the architecture and constraints.
Techniques for mining legacy assets.
Re-coding and re-hosting.
Translators and emulators.
Middleware standards and products.
Management of interfaces.
For a deeper discussion of these characteristics, see “COTS and Legacy Software Integration Issues,”
from the 1998 Ground System Architectures Workshop.
Review of the Verification Adequacy
The requirements described in SWE-028 and SWE-030 cover software verification. Software verification is the software engineering activity that shows confirmation that software products properly reflect the requirements specified for them. Insight into the adequacy of the verification activities includes examination of the following:
Objective evidence of the quality of the product from the software developer (e.g., accompanying documentation, test reports, statistical records, process control).
Inspections and audits of the software.
Reviews of the required documentation.
Reviews of any delegation of verification to the supplier made in the SOW.
When the NASA software team uses periodic reports (e.g., progress reports, or summary and individual test reports) to verify supplier-developed software, the requirements for the data in these reports are specified in the SOW. When software verification will be performed at the supplier's site, the requirements for the method of verification are specified in the SOW.
The NASA software team must be allowed to participate in reviews of the verification activity at the software developer's site. The intent and level of these participation activities are specified in the contract.
Review of Trade Study Data and Results
The team exercising the "insight" role uses the contract-specified access to enable the review of all applicable trade studies performed by the software developer or the project. The performance of adequate trade studies helps assure the correctness of the software development and the software testing choices during the actual software development.
Trade studies compare the relative merits of alternative approaches, and so ensure that the most cost-effective system is developed. They maintain traceability of design decisions back to fundamental requirements. Trade studies do this by comparing alternatives at various levels for the system being developed. They may be applied to concept, design, implementation, verification, support, and other areas. They provide a documented, analytical rationale for choices made in system development.
Trade studies can be used to compare a number of options.
Auditing the Software Development Process
NASA considers a software audit to be an examination of a work product or set of work products performed by a group independent from the developers to assess compliance with specifications, standards, contractual agreements, or other criteria. This NASA definition is from NASA-STD-8739.8 "Software Assurance Standard,"
and is based on IEEE 610.12.
"Software product" mostly, but not exclusively, refers to some kind of technical document. IEEE Std. 1028, IEEE Standard for Software Reviews,
offers a list of 33 "examples of software products subject to audit," including source code and documentary products, such as various sorts of plans, contracts, specifications, designs, procedures, standards, and reports, but also non-documentary products, such as data, test data, and deliverable media.
Software audits are distinct from software peer reviews and software management reviews in that they are conducted by personnel external to, and independent of, the software development organization, and are concerned with compliance of products or processes, rather than with their technical content, technical quality, or managerial implications.
Participation in Software Reviews and Systems and Software Technical Interchange Meetings
The NASA software development team must be afforded the opportunity to attend the regular and periodic meetings that include software development progress reviews, milestone reviews, formal reviews [such as those found in NPR 7123.1
(Systems Engineering), NPR 7120.5
(Space Flight Program and Project Management), NPR 7120.7
(IT and Institutional Infrastructure) and NPR 7120.8
(Research and Technology)], systems engineering meetings regarding software activities, and regular technical interchange meetings. The material reviewed and the ideas for software development that are discussed at these meetings and interchanges often provide a more robust description of the software architecture, the software requirements, the design and discussions of major issues than are typically presented in monthly or quarterly submitted reports.
The meetings, reviews, and technical interchange opportunities must be specified in the SOW. (See Topic topic 7.3 - Acquisition Guidance for advice on oversight activities, specifically what to include for binding reviews, participation on review boards, and how to handle comments and assigned actions.)
If the Defense Contract Management Agency
services are available at the software development site, consider using them for basic monitoring and verification services. Arrangements for these services are typically made by the Center's acquisition department.
4. Small Projects
This requirement applies to all projects regardless of size.
6. Lessons Learned
A documented lesson from the NASA Lessons Learned database notes the following:
Space Shuttle Program/External Tank (ET)/Super Light Weight Tank (SLWT). Lesson Number 1048: A discussion on the shuttle program's certification of its super light weight tank attests to the value of exercising insight and guidance through limited oversight activities
Acquisition and Oversight of Contracted Software Development (1999). Lesson Number 0921: Tailorable acquisition management and oversight processes for NASA contracted software development are essential to ensure that customers receive a quality product. A documented lesson from the NASA Lessons Learned database includes as a cause of the loss of a mission "the lack of a controlled and effective process for acquisition of contractor-developed, mission critical software." In this particular case, the quality of the contractor's product was not monitored as it would have been if the proper milestones for reviewing and auditing contractor progress were in place