Peer Review Metrics

·        # of peer reviews performed vs. # of peer reviews planned

X

X

X

SWE-016

SWE-087

SWE-089

·        # of Non-Conformances identified in each peer review

X

X

X

SWE-087

SWE-089

·        # of Non-Conformances identified by software assurance during each peer review

X

X

X

SWE-087

SWE-088

SWE-089

·        Total # of peer review Non-Conformances (Open, Closed)

X

X

X

SWE-087

SWE-088

SWE-089

·        Preparation time each review participant spent preparing for the review

X

X

X

SWE-088

SWE-089

·        Time required to close review Non-Conformances

X

X

X

SWE-087

SWE-088

SWE-089

·        # of peer review participants vs. total # invited

X

X

X

SWE-088

SWE-089

·        # of peer review Non-Conformances per work product vs. # of peer reviewers

X

X

X

SWE-088

SWE-089

Peer Review Audit Metrics

·        # of audit Non-Conformances per peer review audit

X

X

X

SWE-088

·        # of Peer Review Audits planned vs. # of Peer Review Audits performed

X

X

X

SWE-016

SWE-087

SWE-088

·        Trends on non-conformances from audits (Open, Closed, Life cycle Phase)

X

X

X

SWE-088

SWE-089

·        Time required to close peer review audit Non-Conformances

X

X

X

SWE-088

SWE-089

·        Preparation time each audit participant spent preparing for audit

X

X

X

SWE-088

SWE-089

Problem/Change Report Status Metrics

·        Total # of Non-Conformances over time (Open, Closed, # of days Open, and Severity of Open)

·        # of Non-Conformances in current reporting period (Open, Closed, Severity)

X

X

X

SWE-062

SWE-065c

SWE-065d

SWE-068

SWE-202

SWE-203

SWE-204

·        # of safety-related Non-Conformances

X

X

X

SWE-203

SWE-068

SWE-071

·        Trend of Open vs. Closed Non-Conformances over time

X

X

X

SWE-053

SWE-054

SWE-065

·        Trend of change status over time (# of changes approved, # in implementation, # in test, # closed)

X

X

X

SWE-018

SWE-053

SWE-080

·        # of Non-Conformances identified in embedded COTS, GOT, MOTS, OSS, or reused components in ground or flight software vs. # of Non-Conformances closed

X

SWE-202

SWE-203

SWE-211

·        # of Non-Conformances identified in source code products used (Open, Closed)

X

SWE-202

SWE-203

·        # of software Non-Conformances at each Severity level for each software configuration item (Open, Closed)

Note:  Metrics in bold type are required by all projects

X

X

SWE-202

·        # of Closed action items vs. # of Open action items

X

X

X

SWE-062 SWE-62c
SWE-065d

·        # of Root Cause Analyses performed;

·        # of Non-Conformances identified by each root cause analysis

X

X

SWE-204

SA Corrective Action (CA) metrics (Issues, Risks)

·        # of Corrective Actions (CAs) raised by SA vs. total #

       ·        Attributes (Type, Severity, # of days Open, Life cycle Phase Found)

       ·        State (Open, In work, Closed)

       ·        Trends of CA Open vs. Closures over time

X

X

X

SWE-024

SWE-034

SWE-204

SWE-054

·        Trend the # of inconsistencies or corrective actions identified, and # closed.

X

X

SWE-054

SWE-024

SWE-204

Process Improvement Metrics

·        # of software work product Non-Conformances identified by life cycle phase over time

X

X

X

SWE-013

SWE-022

SWE-024

SWE-039

SWE-051

SWE-054

SWE-057

SWE-058

SWE-062

SWE-065b

SWE-065c

SWE-065d

SWE-068

SWE-071

SWE-075

SWE-079

SWE-084

SWE-086

SWE-087

SWE-125

SWE-134

SWE-139

SWE-146

SWE-157

SWE-159

SWE-184

SWE-185

SWE-187

SWE-191

SWE-194

SWE-201

SWE-204

SWE-205

·        # of software process Non-Conformances by life cycle phase over time

X

X

X

SWE-032

SWE-039

SWE-061

SWE-077

SWE-080

SWE-082

SWE-085

SWE-086

SWE-088

SWE-139

SWE-195

SWE-204

Characteristics Metrics

·        Identify the specific requirements in NASA-STD-8739.8 that are being tailored by the projects (*organizational metric)

X

X

X

SWE-121

SWE-125

SWE-013

SWE-176

·        # of projects tailoring each requirement (*organizational measure)

·        % of requirements tailored per project (*organizational measure)

X

X

SWE-125

SWE-013

SWE-121

·        % of Total Source Code for each Software Classification (*organizational measure)

X

X

SWE-020

SWE-176

SWE-087

Cost/ Effort Metrics

·        Planned SA resource allocation vs. actual SA resource allocation 

X

X

X

SWE-015

SWE-174

SWE-151

·        Comparison of initial SA cost estimates vs. final cost (capturing assumptions and differences)

X

SWE-174

SWE-015

SWE-151

·        Trend SA cost estimates throughout life cycle

X

X

X

SWE-174

Training Metrics

·        % of required training completed for each of the project SA personnel

X

X

SWE-017

·        % of project personnel that have completed project specific training against the planned training schedule

X

X

X

SWE-017

Compliance Audit Metrics

·        # of Compliance Audits planned vs. # of Compliance Audits performed

X

X

X

SWE-024

SWE-039

SWE-139

SWE-016

SWE-032

SWE-077

SWE-195

SWE-082

SWE-084

SWE-085

SWE-086

SWE-088

SWE-079

·        # of Open vs. Closed Audit Non-Conformances over time

·         # of Non-Conformances per audit (including findings from process and compliance audits, process maturity)

X

X

X

SWE-024

SWE-039

SWE-139

SWE-016

SWE-032

SWE-077

SWE-079

SWE-195

SWE-082

SWE-084

SWE-085

SWE-086

SWE-088

SWE-201

·        # of Non-Conformances identified in plans (e.g., SMPs, SDPs, CM Plans, SA Plans, Safety Plans, Test Plans)

X

X

SWE-024

SWE-013

SWE-075

SWE-079

SWE-139

SWE-071

·        #  of Non-Conformances identified in the software Configuration Management Plan

·        Trends of # Open vs. # Closed over time

X

X

X

SWE-079

·        #  of Configuration Management Audits conducted by the project – Planned vs. Actual

X

X

X

SWE-082

SWE-077

SWE-084

·        # of Non-Conformances per audit (including findings from process and compliance audits, process maturity)

X

X

X

SWE-024

SWE-039

SWE-139

SWE-016

SWE-032

SWE-077

SWE-195

SWE-082

SWE-084

SWE-085

SWE-086

SWE-088

SWE-079

·        # of process Non-Conformances (e.g., activities not performed) identified by SA vs. # accepted by the project

·        Trends of # Open vs. # Closed over time

X

X

X

SWE-016

SWE-039

SWE-032

SWE-077

SWE-195

SWE-082

SWE-084

SWE-085

SWE-086

SWE-088

•      # of Non-Conformances found in flight code, ground code, tools, and COTs products used (Open vs. Closed).

X

X

SWE-136

•      # of open non-compliances vs. # of closed non-compliances found in security scans or coding standard compliance audits.

X

X

SWE-63

Project Acceptance Metrics

·        # of Non-Conformances (activities not being performed)

·        # of Non-Conformances accepted by project

·        # of Non-Conformances (Open, Closed, Total)

·        Trends of Open vs, Closed Non-Conformances over time

X

X

X

SWE-087

SWE-201

SWE-039

SWE-139

Progress Tracking Metrics

·        Deviations of actual schedule progress vs. planned schedule progress above defined threshold

X

X

X

SWE-016

SWE-018

SWE-046

·        # of Software Requirements (e.g. Project, Application, Subsystem, System, etc.)

X

X

X

SWE-050

SWE-051

SWE-052

SWE-065b

SWE-066

SWE-194

SWE-027

·        # of Software Requirements that do not trace to a parent requirement   

X

X

SWE-033

SWE-050

SWE-051

·        # of architectural issues identified vs. number closed

SWE-057

SWE-058

SWE-143

·        # of planned unit test cases vs. # of actual unit test cases completed

X

SWE-062

SWE-186

·        Total # of tests completed vs. # of test results evaluated and signed off

X

SWE-066

SWE-068

SWE-065d

SWE-159

·        # of Safety-Critical tests executed vs. # of Safety-Critical tests witnessed by SA

X

X

SWE-066

SWE-068

SWE-065d

SWE-159

SWE-062

SWE-186

·        # of software components (e.g. programs, modules, routines, functions, etc.) planned vs. # actually released in each build

X

SWE-194

SWE-077

SWE-073

SWE-036

·        # of Non-Conformances from reviews (Open vs. Closed; # of days Open)

X

X

X

SWE-134

SWE-037

SWE-039

SWE-087

SWE-088

SWE-089

SWE-143

·        # of Software Requirements being met via satisfactory testing vs. total # of Software Requirements

X

X

SWE-065d

SWE-066

SWE-192

SWE-071

·        # of Software Requirements without associated test cases

X

X

SWE-066

SWE-065b

SWE-071

·      # of  design issues found vs. # of design issues resolved

X

SWE-058

 ·       # of planned units for implementation vs. # of units implemented and unit tested.

X

SWE-060

Risk Management Metrics

·        # of Risks identified in each life cycle phase (Open, Closed)

X

X

X

SWE-086

SWE-039

SWE-179

SWE-154

SWE-156

SWE-190

·        # of Risks by Severity (e.g., red, yellow, green) over time

X

X

X

SWE-032

SWE-033

SWE-039

SWE-086

SWE-154

SWE-156

SWE-179

SWE-190

SWE-191

·        # of Risks with mitigation plans vs. total # of Risks

X

X

X

SWE-032

SWE-033

SWE-039

SWE-086

SWE-154

SWE-156

SWE-179

SWE-190

SWE-191

·        # of Risks trending up over time

·        # of Risks trending down over time

X

X

X

SWE-032

SWE-033

SWE-039

SWE-086

SWE-154

SWE-156

SWE-179

SWE-190

SWE-191

Cybersecurity Risk Metrics

•      # of Cybersecurity vulnerabilities and weaknesses identified
•      # of Cybersecurity vulnerabilities and weaknesses (Open, Closed, Severity)
•      Trending of Open vs. Closed over time

X

X

SWE-159

SWE-135

SWE-063

SWE-201

·        # of Cybersecurity Risks with Mitigations vs. # of Cybersecurity Risks identified

X

X

SWE-154

SWE-156

SWE-151

SWE-159

Traceability Metrics

·        % of traceability completed in each area: System Level requirements to Software requirements; Software Requirements to Design; Design to Code; Software Requirements to Test Procedures

X

X

SWE-052

·        % of traceability completed for all hazards to software requirements and test procedures

X

X

SWE-052

·        Defect trends for trace quality (# of circular traces, orphans, widows, etc.)

X

SWE-051

SWE-052

Requirements Metrics

·        # of detailed software requirements vs. # of estimated SLOC to be developed by the project

X

SWE-050

SWE-051

SWE-151

SWE-174

·        # of incorrect, missing and incomplete requirements (i.e., # of requirements issues) vs. # of requirements issues resolved

X

SWE-051

SWE-053

SWE-054

·        Software Requirements Volatility (# of requirements added, deleted, modified, # of TBDs over time)

Note: Metrics in bold type are required for all projects.

X

X

SWE-200

SWE-053

·        # of TBD/TBC/TBR requirements trended over time

X

X

SWE-066

Safety Metrics

·        # of safety-related requirement issues (Open, Closed) over time

X

X

X

SWE-039

SWE-139

SWE-205

SWE-023

SWE-134

SWE-184

SWE-052

SWE-051

SWE-058

SWE-065b

SWE-066

SWE-071

SWE-192

SWE-080

SWE-087

·        # of safety-related non-conformances identified by life cycle phase (over time, Open vs. Closed, # of days)

X

X

X

SWE-013

SWE-039

SWE-139

SWE-143

SWE-121

SWE-184

SWE-023

SWE-134

SWE-052

SWE-051

SWE-057

SWE-058

SWE-135

SWE-062

SWE-065a

SWE-065b

SWE-065c

SWE-065d

SWE-066

SWE-068

SWE-071

SWE-191

SWE-080

SWE-081

SWE-087

SWE-205

Reuse Metrics

·        # of products submitted for reuse;

·        # of developed products submitted for reuse vs. total # of developed products

X

SWE-147

SWE-148

·        # of developed products entered in NASA Internal Sharing & Reuse System vs. total # of developed products

X

SWE-147

SWE-148

·        # of products submitted for reuse vs. # of products entered into NASA Internal Sharing & Reuse Systems

X

SWE-147

SWE-148

Cybersecurity Metrics

·        # of Cybersecurity vulnerabilities and weaknesses identified

·        # of Cybersecurity vulnerabilities and weaknesses (Open, Closed, Severity)

·        Trending of Open vs. Closed over time

X

X

X

SWE-159

SWE-135

SWE-063

·        # and type of vulnerabilities and weaknesses identified by the project

X

X

SWE-135

SWE-063

·        # of Cybersecurity vulnerabilities and weaknesses identified by life cycle phase

X

X

X

SWE-159

SWE-135

SWE-063

·        # of Cybersecurity vulnerabilities and weaknesses identified vs. # resolved during Implementation

X

X

SWE-159

SWE-135

SWE-063

•      # of requirements specified relating to detection of adversarial actions vs. # of requirements related to detection of adversarial actions actually implemented

X

X

SWE-210

·        # of Non-Conformances identified in Cybersecurity coding standard compliance (Open, Closed)

X

X

SWE-063

SWE-135

SWE-207

Test Coverage Metrics

·         Software code/test coverage percentages for all identified safety-critical components (e.g., # of paths tested vs. total # of possible paths) 

Note: Metrics in bold type are required by all projects

SWE-066
SWE-190
SWE-189 
SWE-219

•      Test coverage data for all identified safety-critical software components.

SWE-219

•      % of code that has been executed during testing

SWE-060

Static Code Metrics

·        Document the Static Code Analysis tools used with associated Non-Conformances

·        # of total errors and warnings identified by the tool

·        # of errors and warnings evaluated vs. # of total errors and warnings identified by the tool

X

SWE-135

SWE-185

·        # of Non-Conformances raised by SA vs. total # of raised Non-Conformances

X

SWE-135

SWE-185

·        # of static code errors and warnings identified as “positives” vs. # of total errors and warnings identified by the tool

·        Total # of static code analysis "positives" vs.  # of "positives" resolved. Trend over time.

·        # of static code errors and warnings resolved by Severity vs. # of static code errors and warnings identified by Severity by the tool

X

SWE-135

SWE-185

·        # of static code “positives” over time (Open, Closed, Severity)

X

SWE-135

SWE-185

·        # of Cybersecurity vulnerabilities and weaknesses identified by the tool

X

SWE-135

SWE-185

·        # of coding standard violations identified (Open, Closed, type of violation, Severity)

X

SWE-061

SWE-185

·        Software cyclomatic complexity # for all identified safety-critical components

X

SWE-087

SWE-134

·        Trend of # of total errors and warnings identified per SCA Tool, Language, and SLOC size

X

SWE-135

·      Total # of static code analysis "positives" vs.  # of "positives" resolved. Trend over time.

X

SWE-135

Test Procedures Metrics

·        #  of Non-Conformances and risks open vs. # of Non-Conformances, risks, identified with test procedures

X

SWE-087

SWE-065b

SWE-071

SWE-191

·        # of hazards with completed test procedures/cases vs. total number of hazards over time

X

SWE-065b

SWE-068

SWE-191

·        # of software requirements with completed test procedures/cases over time

X

SWE-065b

SWE-071

SWE-066

·        # of Non-Conformances identified when the approved, updated requirements are not reflected in test procedures

X

SWE-065b

SWE-071

SWE-052

SWE-066

SWE-191

·        # of Non-Conformances identified while confirming hazard controls are verified through test plans/procedures/cases

X

SWE-065b

SWE-071

SWE-066

SWE-068

SWE-191

•   # of issues and risks/corrective actions open vs. total # of issues and risks/corrective actions identified with test procedures

X

SWE-065b

Test Metrics

·        # of tests executed vs. # of tests completed

X

X

SWE-062

SWE-066

SWE-065d

SWE-068

SWE-159

SWE-190

SWE-192

SWE-055

SWE-080

SWE-191

SWE-194

SWE-211

·        # of Non-Conformances identified during each testing phase (Open, Closed, Severity)

X

X

SWE-062

SWE-065d

SWE-066

SWE-068

SWE-080

SWE-159

SWE-190

SWE-191

SWE-192

SWE-194

SWE-211

·        # of Requirements tested vs. total # of Requirements

X

X

SWE-062

SWE-065b

SWE-065d

SWE-066

SWE-080

SWE-191

SWE-192

SWE-194

·        # of Hazards containing software that have been tested vs. total # of Hazards containing software

X

X

SWE-062

SWE-066

SWE-068

SWE-080

SWE-134

SWE-192

SWE-205

·        # of Non-Conformances identified in models, simulations, and tools over time (Open, Closed, Severity)

X

X

SWE-070

·        # of Regression test set Non-Conformances/Risks over time (Open, Closed, Severity)

X

X

SWE-191

·        # of Requirements tested in customer environment vs. # of Requirements

X

SWE-055

Cybersecurity Metrics During Testing Metrics

·        # of Cybersecurity mitigation implementations identified from the security vulnerabilities and security weaknesses

X

X

SWE-154

SWE-159

SWE-191

·        # of Cybersecurity mitigation implementations identified with associated test procedures vs. # of Cybersecurity mitigation implementations identified

X

X

SWE-154

SWE-159

SWE-191

·        # of Cybersecurity mitigation tests completed vs. total # of Cybersecurity mitigation tests

X

X

SWE-159

·        # of Non-Conformances identified during Cybersecurity mitigation testing (Open, Closed, Severity)

X

X

SWE-159

·        Trends of Cybersecurity Non-Conformances over time

X

X

SWE-159

SWE-063

SWE-135

SWE-201

Test Coverage Metrics

·        Software code/test coverage percentages for all identified safety-critical components (e.g., # of paths tested vs. total # of possible paths)

X

X

SWE-066

SWE-190

SWE-134

SWE-189

·        # of tests completed vs. total # of tests

X

X

SWE-055

SWE-062

SWE-066

SWE-065d

SWE-068

SWE-080

SWE-159

SWE-190

SWE-191

SWE-192

SWE-194

SWE-211

·        # of detailed software requirements tested to date vs. total # of detailed software requirements

X

X

SWE-055

SWE-062

SWE-065b

SWE-065d

SWE-066

SWE-071

SWE-080

SWE-191

SWE-192

SWE-194

SWE-159

·        # of safety-critical requirement verifications vs. total # of safety-critical requirement verifications completed

·        # of Open issues vs. # of Closed over time

X

X

SWE-062

SWE-066

SWE-191

SWE-192

SWE-068

·        # of Source Lines of Code (SLOC) tested vs. total # of SLOC

X

X

SWE-066

SWE-134

SWE-189

SWE-190

SWE-219

Build/Release Content Metrics

·        # of software units planned vs. # actually built

X

X

SWE-063

SWE-194

·        # of planned software requirements implemented in each build vs. # of actual software requirements implemented in each build

X

X

SWE-063

SWE-194

·        # of Non-Conformances identified in release documentation (Open, Closed)

X

X

SWE-063

SWE-077

SWE-084

SWE-085

Operations/ Maintenance Metrics

·        # of Non-Conformances identified in the software after delivery

X

SWE-195

Other Metrics

•      Measures relating to status and performance as identified in other requirements. (Schedule deviations, closure of corrective actions, product and process audit results, peer review result, etc)

X

X

SWE-093

•      # of requirements being met, any TBD/TBC/TBR requirements, or any requirements that are not being met

•      Performance and functionality measures (Schedule deviations, closure of corrective actions, product and process audit results, peer review results, requirements volatility, # of requirements satisfactorily tested vs. total # of requirements, etc.)

X

X

SWE-199

•      # of findings open vs. # of findings identified

SWE-082

•   Trends of Open vs. Closed Non-Conformances change reports over time.

SWE-065d

IV&V Metrics (Kept by IV&V)

·        # of IV&V Non-Conformances/issues,/risks (Open, Closed, accepted by the project, severity, category of Non-Conformance (e.g., requirements severity, category (requirements, design, code, test, documentation, etc.))

·        Kept by IV&V team, not SA

X

X

X

SWE-179