8.18 - SA Suggested Metrics - Table with no macros

Peer Review Metrics

# of peer reviews performed vs. # of peer reviews planned

X

X

X

SWE-016

SWE-087

SWE-089

Peer Review Metrics

# of Non-Conformances identified in each peer review

X

X

X

SWE-087

SWE-089

Peer Review Metrics

# of Non-Conformances identified by software assurance during each peer review

X

X

X

SWE-087

SWE-088

SWE-089

Peer Review Metrics

Total # of peer review Non-Conformances (Open, Closed)

X

X

X

SWE-087

SWE-088

SWE-089

Peer Review Metrics

Preparation time each review participant spent preparing for the review

X

X

X

SWE-088

SWE-089

Peer Review Metrics

Time required to close review Non-Conformances

X

X

X

SWE-087

SWE-088

SWE-089

Peer Review Metrics

# of peer review participants vs. total # invited

X

X

X

SWE-088

SWE-089

Peer Review Metrics

# of peer review Non-Conformances per work product vs. # of peer reviewers

X

X

X

SWE-088

SWE-089

Peer Review Audit Metrics

# of audit Non-Conformances per peer review audit

X

X

X

SWE-088

Peer Review Audit Metrics

# of Peer Review Audits planned vs. # of Peer Review Audits performed

X

X

X

SWE-016

SWE-087

SWE-088

Peer Review Audit Metrics

Trends on non-conformances from audits (Open, Closed, Life cycle Phase)

X

X

X

SWE-088

SWE-089

Peer Review Audit Metrics

Time required to close peer review audit Non-Conformances

X

X

X

SWE-088

SWE-089

Peer Review Audit Metrics

Preparation time each audit participant spent preparing for audit

X

X

X

SWE-088

SWE-089

Problem/Change Report Status Metrics

Total # of Non-Conformances over time (Open, Closed, # of days Open, and Severity of Open)

# of Non-Conformances in current reporting period (Open, Closed, Severity)

X

X

X

SWE-062

SWE-065c

SWE-065d

SWE-068

SWE-202

SWE-203

SWE-204

Problem/Change Report Status Metrics

# of safety-related Non-Conformances

X

X

X

SWE-203

SWE-068

SWE-071

Problem/Change Report Status Metrics

Trend of Open vs. Closed Non-Conformances over time

X

X

X

SWE-053

SWE-054

SWE-065

Problem/Change Report Status Metrics

Trend of change status over time (# of changes approved, # in implementation, # in test, # closed)

X

X

X

SWE-018

SWE-053

SWE-080

Problem/Change Report Status Metrics

# of Non-Conformances identified in embedded COTS, GOT, MOTS, OSS, or reused components in ground or flight software vs. # of Non-Conformances closed

X

SWE-136

SWE-202

SWE-203

SWE-211

Problem/Change Report Status Metrics

# of Non-Conformances identified in source code products used (Open, Closed)

X

SWE-202

SWE-203

Problem/Change Report Status Metrics

# of software Non-Conformances at each Severity level for each software configuration item (Open, Closed)

        Bold type indicates a required item

X

X

SWE-202

Problem/Change Report Status Metrics

# of Closed action items vs. # of Open action items

X

X

X

SWE-062 SWE-62c

SWE-065d

Problem/Change Report Status Metrics

# of Root Cause Analyses performed;

# of Non-Conformances identified by each root cause analysis

X

X

SWE-204

SA Corrective Action (CA) metrics (Issues, Risks)

# of Corrective Actions (CAs) raised by SA vs. total #

       Attributes (Type, Severity, # of days Open, Life cycle Phase Found)

      State (Open, In work, Closed)

      Trends of CA Open vs. Closures over time

X

X

X

SWE-024

SWE-204

SWE-054

SA Corrective Action (CA) metrics (Issues, Risks)

Trend the # of inconsistencies or corrective actions identified, and # closed.

X

X

SWE-054

SWE-024

SWE-204

Process Improvement Metrics

# of software work product Non-Conformances identified by life cycle phase over time

X

X

X

SWE-013

SWE-022

SWE-024

SWE-039

SWE-051

SWE-054

SWE-057

SWE-058

SWE-062

SWE-065b

SWE-065c

SWE-065d

SWE-068

SWE-071

SWE-075

SWE-079

SWE-084

SWE-086

SWE-087

SWE-125

SWE-134

SWE-139

SWE-146

SWE-157

SWE-159

SWE-184

SWE-185

SWE-187

SWE-191

SWE-194

SWE-201

SWE-204

SWE-205

Process Improvement Metrics

# of software process Non-Conformances by life cycle phase over time

X

X

X

SWE-032

SWE-039

SWE-061

SWE-077

SWE-080

SWE-082

SWE-085

SWE-086

SWE-088

SWE-139

SWE-195

SWE-204

Process Improvement Metrics

Identify the specific requirements in NASA-STD-8739.8 that are being tailored by the projects (*organizational metric)

X

X

X

SWE-121

SWE-125

SWE-013

SWE-176

Process Improvement Metrics

# of projects tailoring each requirement (*organizational measure)

% of requirements tailored per project (*organizational measure)

X

X

SWE-125

SWE-013

SWE-121

Process Improvement Metrics

% of Total Source Code for each Software Classification (*organizational measure)

X

X

SWE-020

SWE-176

SWE-087

Cost/ Effort Metrics

Planned SA resource allocation vs. actual SA resource allocation 

X

X

X

SWE-015

SWE-174

SWE-151

Cost/ Effort Metrics

Comparison of initial SA cost estimates vs. final cost (capturing assumptions and differences)

X

SWE-174

SWE-015

SWE-151

Cost/ Effort Metrics

Trend SA cost estimates throughout life cycle

X

X

X

SWE-174

Training Metrics

% of required training completed for each of the project SA personnel

X

X

SWE-017

Training Metrics

% of project personnel that have completed project specific training against the planned training schedule

X

X

X

SWE-017

Compliance Audit Metrics

# of Compliance Audits planned vs. # of Compliance Audits performed

X

X

X

SWE-024

SWE-039

SWE-139

SWE-016

SWE-032

SWE-077

SWE-195

SWE-082

SWE-084

SWE-085

SWE-086

SWE-088

SWE-079 SWE-201

Compliance Audit Metrics

# of Open vs. Closed Audit Non-Conformances over time

# of Non-Conformances per audit (including findings from process and compliance audits, process maturity)

X

X

X

SWE-022

SWE-024

SWE-039

SWE-139

SWE-016

SWE-032

SWE-077

SWE-079

SWE-195

SWE-082

SWE-084

SWE-085

SWE-086

SWE-088

SWE-201

Compliance Audit Metrics

# of Non-Conformances identified in plans (e.g., SMPs, SDPs, CM Plans, SA Plans, Safety Plans, Test Plans)

X

X

SWE-024

SWE-013

SWE-075

SWE-079

SWE-139

SWE-071

Compliance Audit Metrics

#  of Non-Conformances identified in the software Configuration Management Plan

Trends of # Open vs. # Closed over time

X

X

X

SWE-079

Compliance Audit Metrics

#  of Configuration Management Audits conducted by the project – Planned vs. Actual

X

X

X

SWE-082

SWE-077

SWE-084

Compliance Audit Metrics

# of Non-Conformances per audit (including findings from process and compliance audits, process maturity)

X

X

X

SWE-024

SWE-039

SWE-139

SWE-016

SWE-032

SWE-077

SWE-195

SWE-082

SWE-084

SWE-085

SWE-086

SWE-088

SWE-079

Compliance Audit Metrics

# of process Non-Conformances (e.g., activities not performed) identified by SA vs. # accepted by the project

Trends of # Open vs. # Closed over time

X

X

X

SWE-016

SWE-039

SWE-032

SWE-077

SWE-195

SWE-082

SWE-084

SWE-085

SWE-086

SWE-088

Compliance Audit Metrics

# of Non-Conformances found in flight code, ground code, tools, and COTs products used (Open vs. Closed).

X

X

SWE-136

Compliance Audit Metrics

# of open non-compliances vs. # of closed non-compliances found in security scans or coding standard compliance audits.

X

X

SWE-63

Project Acceptance Metrics

# of Non-Conformances (activities not being performed)

# of Non-Conformances accepted by project

# of Non-Conformances (Open, Closed, Total)

Trends of Open vs, Closed Non-Conformances over time

X

X

X

SWE-087

SWE-201

SWE-039

SWE-139

Progress Tracking Metrics

Deviations of actual schedule progress vs. planned schedule progress above defined threshold

X

X

X

SWE-016

SWE-018

SWE-046

Progress Tracking Metrics

# of Software Requirements (e.g. Project, Application, Subsystem, System, etc.)

X

X

X

SWE-050

SWE-051

SWE-052

SWE-065b

SWE-066

SWE-194

SWE-027

Progress Tracking Metrics

# of Software Requirements that do not trace to a parent requirement   

X

X

SWE-033

SWE-050

SWE-051

Progress Tracking Metrics

# of architectural issues identified vs. number closed

SWE-057

SWE-058

SWE-143

Progress Tracking Metrics

# of planned unit test cases vs. # of actual unit test cases completed

X

SWE-062

SWE-186

Progress Tracking Metrics

Total # of tests completed vs. # of test results evaluated and signed off

X

SWE-066

SWE-068

SWE-065d

SWE-159

Progress Tracking Metrics

# of Safety-Critical tests executed vs. # of Safety-Critical tests witnessed by SA

X

X

SWE-066

SWE-068

SWE-065d

SWE-159

SWE-062

SWE-186

Progress Tracking Metrics

# of software components (e.g. programs, modules, routines, functions, etc.) planned vs. # actually released in each build

X

SWE-194

SWE-077

SWE-073

SWE-036

Progress Tracking Metrics

# of Non-Conformances from reviews (Open vs. Closed; # of days Open)

X

X

X

SWE-134

SWE-037

SWE-039

SWE-087

SWE-088

SWE-089

SWE-143

Progress Tracking Metrics

# of Software Requirements being met via satisfactory testing vs. total # of Software Requirements

X

X

SWE-065d

SWE-066

SWE-192

SWE-071

Progress Tracking Metrics

# of Software Requirements without associated test cases

X

X

SWE-066

SWE-065b

SWE-071

Progress Tracking Metrics

# of  design issues found vs. # of design issues resolved

X

SWE-058

Progress Tracking Metrics

 # of planned units for implementation vs. # of units implemented and unit tested.

X

SWE-060

Risk Management Metrics

# of Risks identified in each life cycle phase (Open, Closed)

X

X

X

SWE-086

SWE-039

SWE-179

SWE-154

SWE-156

SWE-190

Risk Management Metrics

# of Risks by Severity (e.g., red, yellow, green) over time

X

X

X

SWE-032

SWE-033

SWE-039

SWE-086

SWE-154

SWE-156

SWE-179

SWE-190

SWE-191

Risk Management Metrics

# of Risks with mitigation plans vs. total # of Risks

X

X

X

SWE-032

SWE-033

SWE-039

SWE-086

SWE-154

SWE-156

SWE-179

SWE-190

SWE-191

Risk Management Metrics

# of Risks trending up over time

# of Risks trending down over time

X

X

X

SWE-032

SWE-033

SWE-039

SWE-086

SWE-154

SWE-156

SWE-179

SWE-190

SWE-191

Cybersecurity Risk Metrics

# of Cybersecurity Risks identified (Open, Closed, Severity)

X

X

SWE-154

SWE-156

Cybersecurity Risk Metrics

·of Cybersecurity Risks with Mitigations vs. # of Cybersecurity Risks identified

X

X

SWE-154

SWE-156

SWE-159

Traceability Metrics

% of traceability completed in each area: System Level requirements to Software requirements; Software Requirements to Design; Design to Code; Software Requirements to Test Procedures

X

X

SWE-052

Traceability Metrics

% of traceability completed for all hazards to software requirements and test procedures

X

X

SWE-052

Traceability Metrics

Defect trends for trace quality (# of circular traces, orphans, widows, etc.)

X

SWE-051

SWE-052

Requirements Metrics

# of detailed software requirements vs. # of estimated SLOC to be developed by the project

X

SWE-050

SWE-051

SWE-151

SWE-174

Requirements Metrics

# of incorrect, missing and incomplete requirements (i.e., # of requirements issues) vs. # of requirements issues resolved

X

SWE-051

SWE-053

SWE-054

Requirements Metrics

Software Requirements Volatility (# of requirements added, deleted, modified, # of TBDs over time)

X

X

SWE-200

SWE-053

Requirements Metrics

# of TBD/TBC/TBR requirements trended over time

X

X

SWE-066

Safety Metrics

# of safety-related requirement issues (Open, Closed) over time

X

X

X

SWE-039

SWE-139

SWE-205

SWE-023

SWE-134

SWE-184

SWE-052

SWE-051

SWE-058

SWE-065b

SWE-066

SWE-071

SWE-192

SWE-080

SWE-087

Safety Metrics

# of safety-related non-conformances identified by life cycle phase (over time, Open vs. Closed, # of days)

X

X

X

SWE-013

SWE-039

SWE-139

SWE-143

SWE-121

SWE-184

SWE-023

SWE-134

SWE-052

SWE-051

SWE-057

SWE-058

SWE-135

SWE-062

SWE-065a

SWE-065b

SWE-065c

SWE-065d

SWE-066

SWE-068

SWE-071

SWE-191

SWE-080

SWE-081

SWE-087

SWE-205

Reuse Metrics

# of products submitted for reuse;

# of developed products submitted for reuse vs. total # of developed products

X

SWE-147

SWE-148

Reuse Metrics

# of developed products entered in NASA Internal Sharing & Reuse System vs. total # of developed products

X

SWE-147

SWE-148

Reuse Metrics

# of products submitted for reuse vs. # of products entered into NASA Internal Sharing & Reuse Systems

X

SWE-147

SWE-148

Cybersecurity Metrics

# of Cybersecurity vulnerabilities and weaknesses identified

# of Cybersecurity vulnerabilities and weaknesses (Open, Closed, Severity)

Trending of Open vs. Closed over time

X

X

X

SWE-159

SWE-135

SWE-063

Cybersecurity Metrics

# and type of vulnerabilities and weaknesses identified by the project

X

X

SWE-135

SWE-063

Cybersecurity Metrics

# of Cybersecurity vulnerabilities and weaknesses identified by life cycle phase

X

X

X

SWE-159

SWE-135

SWE-063

Cybersecurity Metrics

# of Cybersecurity vulnerabilities and weaknesses identified vs. # resolved during Implementation

X

X

SWE-159

SWE-135

SWE-063

Cybersecurity Metrics

# of requirements specified relating to detection of adversarial actions vs. # of requirements related to detection of adversarial actions actually implemented

X

X

SWE-210

Cybersecurity Metrics

# of Non-Conformances identified in Cybersecurity coding standard compliance (Open, Closed)

X

X

SWE-063

SWE-135

SWE-207

Test Coverage Metrics

Software code/test coverage percentages for all identified safety-critical components (e.g., # of paths tested vs. total # of possible paths) 

Note: Metrics in bold type are required by all projects

SWE-066
SWE-190
SWE-189 
SWE-219

Test Coverage Metrics

Test coverage data for all identified safety-critical software components.

SWE-219

Test Coverage Metrics

% of code that has been executed during testing

SWE-060

Static Code Metrics

Document the Static Code Analysis tools used with associated Non-Conformances

·of total errors and warnings identified by the tool

# of errors and warnings evaluated vs. # of total errors and warnings identified by the tool

X

SWE-135

SWE-185

Static Code Metrics

# of Non-Conformances raised by SA vs. total # of raised Non-Conformances

X

SWE-135

SWE-185

Static Code Metrics

# of static code errors and warnings identified as “positives” vs. # of total errors and warnings identified by the tool

Total # of static code analysis "positives" vs.  # of "positives" resolved. Trend over time.

# of static code errors and warnings resolved by Severity vs. # of static code errors and warnings identified by Severity by the tool

X

SWE-135

SWE-185

Static Code Metrics

# of static code “positives” over time (Open, Closed, Severity)

X

SWE-135

SWE-185

Static Code Metrics

# of Cybersecurity vulnerabilities and weaknesses identified by the tool

X

SWE-135

SWE-185

Static Code Metrics

# of coding standard violations identified (Open, Closed, type of violation, Severity)

X

SWE-061

SWE-185

Static Code Metrics

Software cyclomatic complexity # for all identified safety-critical components

X

SWE-087

SWE-134

Static Code Metrics

Trend of # of total errors and warnings identified per SCA Tool, Language, and SLOC size

X

SWE-135

Static Code Metrics

Total # of static code analysis "positives" vs.  # of "positives" resolved. Trend over time.

X

SWE-135

Test Procedures Metrics

#  of Non-Conformances and risks open vs. # of Non-Conformances, risks, identified with test procedures

X

SWE-087

SWE-065b

SWE-071

SWE-191

Test Procedures Metrics

# of hazards with completed test procedures/cases vs. total number of hazards over time

X

SWE-065b

SWE-068

SWE-191

Test Procedures Metrics

# of software requirements with completed test procedures/cases over time

X

SWE-065b

SWE-071

SWE-066

Test Procedures Metrics

# of Non-Conformances identified when the approved, updated requirements are not reflected in test procedures

X

SWE-065b

SWE-071

SWE-052

SWE-066

SWE-191

Test Procedures Metrics

# of Non-Conformances identified while confirming hazard controls are verified through test plans/procedures/cases

X

SWE-065b

SWE-071

SWE-066

SWE-068

SWE-191

Test Procedures Metrics

# of issues and risks/corrective actions open vs. total # of issues and risks/corrective actions identified with test procedures

X

SWE-065b

Test Metrics

# of tests executed vs. # of tests completed

X

X

SWE-062

SWE-066

SWE-065d

SWE-068

SWE-159

SWE-190

SWE-192

SWE-055

SWE-080

SWE-191

SWE-194

SWE-211

Test Metrics

# of Non-Conformances identified during each testing phase (Open, Closed, Severity)

X

X

SWE-062

SWE-065d

SWE-066

SWE-068

SWE-080

SWE-159

SWE-190

SWE-191

SWE-192

SWE-194

SWE-211

Test Metrics

·of Requirements tested vs. total # of Requirements

X

X

SWE-062

SWE-065b

SWE-065d

SWE-066

SWE-080

SWE-191

SWE-192

SWE-194

Test Metrics

# of Hazards containing software that have been tested vs. total # of Hazards containing software

X

X

SWE-062

SWE-066

SWE-068

SWE-080

SWE-134

SWE-192

SWE-205

Test Metrics

# of Non-Conformances identified in models, simulations, and tools over time (Open, Closed, Severity)

X

X

SWE-070

Test Metrics

# of Regression test set Non-Conformances/Risks over time (Open, Closed, Severity)

X

X

SWE-191

Test Metrics

# of Requirements tested in customer environment vs. # of Requirements

X

SWE-055

Cybersecurity Metrics During Testing Metrics

# of Cybersecurity mitigation implementations identified from the security vulnerabilities and security weaknesses

X

X

SWE-154

SWE-159

SWE-191

Cybersecurity Metrics During Testing Metrics

# of Cybersecurity mitigation implementations identified with associated test procedures vs. # of Cybersecurity mitigation implementations identified

X

X

SWE-154

SWE-159

SWE-191

Cybersecurity Metrics During Testing Metrics

# of Cybersecurity mitigation tests completed vs. total # of Cybersecurity mitigation tests

X

X

SWE-159

Cybersecurity Metrics During Testing Metrics

# of Non-Conformances identified during Cybersecurity mitigation testing (Open, Closed, Severity)

X

X

SWE-159

Cybersecurity Metrics During Testing Metrics

Trends of Cybersecurity Non-Conformances over time

X

X

SWE-159

SWE-063

SWE-135

SWE-201

Test Coverage Metrics

Software code/test coverage percentages for all identified safety-critical components (e.g., # of paths tested vs. total # of possible paths)

X

X

SWE-066

SWE-190

SWE-134

SWE-189

Test Coverage Metrics

# of tests completed vs. total # of tests

X

X

SWE-055

SWE-062

SWE-066

SWE-065d

SWE-068

SWE-080

SWE-159

SWE-190

SWE-191

SWE-192

SWE-194

SWE-211

Test Coverage Metrics

# of detailed software requirements tested to date vs. total # of detailed software requirements

X

X

SWE-055

SWE-062

SWE-065b

SWE-065d

SWE-066

SWE-071

SWE-080

SWE-191

SWE-192

SWE-194

SWE-159

Test Coverage Metrics

# of safety-critical requirement verifications vs. total # of safety-critical requirement verifications completed

# of Open issues vs. # of Closed over time

X

X

SWE-062

SWE-066

SWE-191

SWE-192

SWE-068

Test Coverage Metrics

# of Source Lines of Code (SLOC) tested vs. total # of SLOC

X

X

SWE-066

SWE-134

SWE-189

SWE-190

SWE-219

Build/Release Content Metrics

# of software units planned vs. # actually built

X

X

SWE-063

SWE-194

Build/Release Content Metrics

# of planned software requirements implemented in each build vs. # of actual software requirements implemented in each build

X

X

SWE-063

SWE-194

Build/Release Content Metrics

# of Non-Conformances identified in release documentation (Open, Closed)

X

X

SWE-063

SWE-077

SWE-084

SWE-085

Operations/ Maintenance Metrics

# of Non-Conformances identified in the software after delivery

X

SWE-195

Other Metrics

Measures relating to status and performance as identified in other requirements. (Schedule deviations, closure of corrective actions, product and process audit results, peer review result, etc)

X

X

SWE-093

Other Metrics

# of requirements being met, any TBD/TBC/TBR requirements, or any requirements that are not being met

Performance and functionality measures (Schedule deviations, closure of corrective actions, product and process audit results, peer review results, requirements volatility, # of requirements satisfactorily tested vs. total # of requirements, etc.)

X

X

SWE-199

Other Metrics

# of findings open vs. # of findings identified

X

X

SWE-082

Other Metrics

Trends of Open vs. Closed Non-Conformances change reports over time.

X

X

SWE-065d

IV&V Metrics (Kept by IV&V)

# of IV&V Non-Conformances/issues,/risks (Open, Closed, accepted by the project, severity, category of Non-Conformance (e.g., requirements severity, category (requirements, design, code, test, documentation, etc.))

Kept by IV&V team, not SA

X

X

X

SWE-179

Coverage Metrics

Code coverage data: % of code that has been executed during testing

X

X

SWE-060

Coverage Metrics

Test coverage data for all identified safety-critical software components.

 X

X

SWE-219