4. Small Projects

No additional guidance is available for small projects.

5. Resources

5.1 References

5.2 Tools

6. Lessons Learned

6.1 NASA Lessons Learned

The NASA Lessons Learned database contains the following lessons learned related to ensuring plans and products remain aligned with requirements:

• Risk Assessment in Software Development Projects (Uncertainty caused by changing requirements.) Lesson Number 1321

  Swerefn
  refnum 549

  : "Even with expert and experienced programmers, each new software program presents new technical challenges. Changing methodology and requirements during the design phase of a software project adds uncertainty to the project."
• Deficiencies in Mission Critical Software Development for Mars Climate Orbiter (1999) (Inconsistency led to the loss of mission.) Lesson Number 0740

  Swerefn
  refnum 521

  : "The root cause of the MCO mission loss was an error in the "Sm_forces" program output files, which were delivered to the navigation team in English units (pounds-force seconds) instead of the specified metric units (Newton-seconds). Comply with preferred software review practices, identify mission-critical software (for which staff must participate in major design reviews, walkthroughs, and review of acceptance test results), train personnel in software walkthroughs, and verify consistent engineering units on all parameters.

6.2 Other Lessons Learned

No other Lessons Learned have currently been identified for this requirement.

7. Software Assurance

7.1 Tasking for Software Assurance

7.2 Software Assurance Products

• List of any inconsistencies identified among products, including risks and issues

  
  

  Note
  title Objective Evidence
  Document change management system results. Software problem reporting or defect tracking system results.

  Expand
  title Definition of objective evidence
  Include Page SITE:Definition of Objective Evidence SITE:Definition of Objective Evidence

7.3 Metrics

• # of Corrective Actions (CAs) raised by SA vs. total #
  • Attributes (Type, Severity, # of days Open, Life Cycle Phase Found)
  • State (Open, In work, Closed)
  • Trends of CA Open vs. Closures over time
• # of incorrect, missing, and incomplete requirements (i.e., # of requirements issues) vs. # of requirements issues resolved
• Trend the # of inconsistencies or corrective actions identified, and # closed.
• # of software work product Non-Conformances identified by life cycle phase over time
• Trend of Open vs. Closed Non-Conformances over time

7.4 Guidance

1. Analyze the different project planning documents and confirm that all inconsistencies are documented and addressed. Items that are often overlooked include some of the following:

• A set of requirements with estimated effort too great for the allotted time and effort planned in the schedule (Also, budget and schedule mismatch with requirements)
• Items planned for implementation or purchase too late for the "need" date for integration
• Development schedules not aligned with "need" dates
• Assurance activities planned that don't align with project phases or activities
• Requirements needing expertise not planned or coordination with another group or development that has not been planned

Any inconsistencies identified during the SA review of project plans, requirements, and software products should be documented along with suggested corrective actions, submitted to the project management, and tracked to closure. As the project progresses, inconsistencies among different project products should be kept in mind, particularly during reviews and meetings, and continually identified and addressed as needed

2. Software assurance should analyze all proposed changes for impacts, looking closely at any impacts the change may have on any of the software related to safety or security. The analysis should also consider whether there will be any impacts on existing interfaces or the use of any COTS, GOTS, MOTS, or reused software in the system and whether the change will impact any future maintenance effort. Any identified risks should be brought up to discuss the approval/rejection of the change.

Examples:

• Ensure that the software requirements are met by the design and code.
• Ensure that the planned activities in the project software plans are followed and completed
• Ensure that the software products meet the requirements, conform to the software plans, and meet the acceptance criteria 
• Ensure that the software code meets the software requirements and does not contain features and capabilities that are not included in the requirements, make sure that all software features and capabilities are tested.

3. Confirm:

• That the project tracks the changes

Software assurance will check to see that any changes that are submitted are properly documented and tracked through all the states of resolution (investigation, acceptance/rejection, implementation, test, closure) in the project tracking system.

• That the changes are approved and documented before implementation

Software assurance should track the changes from their submission to their closure or rejection. Initially, SA should confirm that all changes follow the change management process that the project has established. Initially, the change will be documented and submitted to the authorizing CCB for consideration. The authorizing CCB (which will include a software assurance person) will evaluate any changes for impacts.

If the software is safety-critical, the responsible software assurance personnel will perform a software safety change analysis to evaluate whether the proposed change could invoke a hazardous state, affect a control for a hazard, condition, or state, increase the likelihood or severity of a hazardous state, adversely affect safety-critical software, or change the safety criticality of an existing software element.  Keep in mind that changes to the hardware or the software can impact the overall system’s safety and while the focus is on software changes, the software also needs to be aware of changes to the hardware that may impact how software controls, monitors and analyzes inputs from that hardware.  Hardware and software changes can alter the role of software from non-safety-critical to safety-critical or change the severity from moderate to critical. 

Some other considerations for the evaluation of changes:

• • • Is the change an error correction or a new requirement?
    • Will the change fix the problem without major changes to other areas?
    • If major changes to other areas are needed, are they specified, and is this change really necessary?
    • If the change is a requirements change, has the new requirement been approved?
    • How much effort will be required to implement the change?
    • If there is an impact on safety or reliability, are there additional changes that need to be made in those areas? Note: If there is a conflict between safety and security, safety changes have priority.

When all the impacts are considered, the CCB votes on acceptance/rejection. Software assurance is a voting member of the CCB.  Software assurance verifies that the decision is recorded and is acceptable, defined as:

• When the resolution is to “accept as is”, verify that the impact of that resolution on quality, safety, reliability, and security is compatible with the Project’s risk posture and is compliant with NPR 7150.2 and other Center and Agency requirements for risk.
• When the resolution is a change to the SW, the change will sufficiently address the problem and will not impact quality, safety, reliability, security, and compliance with NPR 7150.2; the change will not introduce new or exacerbate other, discrepancies or problems.
• In either case, the presence of other instances of the same kind of discrepancy/problem has been sought out and, if detected, addressed accordingly.
• Verify that appropriate software severity levels are assigned and maintained.
• Assure any risk associated with the change is added to the Project/facility risk management system and is addressed, as needed, in safety, reliability, or other risk systems
• That the implementation of the changes is complete

Software assurance will check to see if the implementation of the approved changes has been coded as per the change request. Check to see that any associated documentation changes are submitted/approved and/or made as needed (i.e., updates to requirements, design, test plans/procedures, etc.)

• That the project tests the changes

Software assurance will check to see that the project test any of the code that has changed and runs a set of regression tests to see that the change has not caused a problem anywhere else in the software system. If the software is safety-critical, a full set of regression tests should be run to ensure that there was no impact on the safety-critical functions.

4. Confirm software changes are done in the software control process

Software assurance will check that the software control process has been followed throughout the handling of the submitted change and that the status of the change is  recorded and confirmed as closed

7.5 Additional Guidance

Additional guidance related to

this requirement may be found in the following

materials in this

Handbook: