Final Report for Software Service History and Airborne Electronic Hardware Service Experience in ...

195 Pages·2016·2.18 MB·English

by  Mutuel, Laurence

DOT/FAA/TC-16/18 Final Report for

Federal Aviation Administration Software Service History and
William J. Hughes Technical Center
Aviation Research Division Airborne Electronic Hardware
Atlantic City International Airport
New Jersey 08405
Service Experience in Airborne

Systems

November 2016

Final Report

This document is available to the U.S. public
through the National Technical Information
Services (NTIS), Springfield, Virginia 22161.

This document is also available from the
Federal Aviation Administration William J. Hughes
Technical Center at actlibrary.tc.faa.gov.

U.S. Department of Transportation
Federal Aviation Administration
NOTICE

This document is disseminated under the sponsorship of the U.S.
Department of Transportation in the interest of information exchange. The
U.S. Government assumes no liability for the contents or use thereof. The
U.S. Government does not endorse products or manufacturers. Trade or
manufacturers’ names appear herein solely because they are considered
essential to the objective of this report. The findings and conclusions in
this report are those of the author(s) and do not necessarily represent the
views of the funding agency. This document does not constitute FAA
policy. Consult the FAA sponsoring organization listed on the Technical
Documentation page as to its use.

This report is available at the Federal Aviation Administration William J.
Hughes Technical Center’s Full-Text Technical Reports page:
actlibrary.tc.faa.gov in Adobe Acrobat portable document format (PDF).
Technical Report Documentation Page
1. Report No. 2. Government Accession No. 3. Recipient's Catalog No.
DOT/FAA/TC-16/18

4. Title and Subtitle 5. Report Date

FINAL REPORT FOR SOFTWARE SERVICE HISTORY AND AIRBORNE November 2016
ELECTRONIC HARDWARE SERVICE EXPERIENCE IN AIRBORNE 6. Performing Organization Code

SYSTEMS

7. Author(s) 8. Performing Organization Report No.

Laurence H. Mutuel, Franck Bayle, Franck Aime D8
9. Performing Organization Name and Address 10. Work Unit No. (TRAIS)

Thales Avionics, Inc.
2733 South Crystal Drive, Suite 1200 11. Contract or Grant No.

Arlington, VA 22202
DTFACT-13-D-00008
12. Sponsoring Agency Name and Address 13. Type of Report and Period Covered

Federal Aviation Administration Final Report
William J. Hughes Technical Center 14. Sponsoring Agency Code

Aviation Research Division
AIR-130

Atlantic City International Airport, NJ 08405
15. Supplementary Notes

The Federal Aviation Administration William J. Hughes Technical Center Aviation Research Division Technical Monitor was
Srini Mandalapu.
16. Abstract
RTCA DO-178C “Software Considerations in Airborne Systems and Equipment Certification” and RTCA DO-254 “Design
Assurance Guidance for Airborne Electronic Hardware” acknowledge, respectively, that acceptable justification software service
history and airborne electronic hardware (AEH) service experience may be used as an acceptable alternative to satisfy one or
several safety objectives. The selected approach for using such data is to provide an equivalent level of confidence in the software
or AEH maturity (or stability) to claim certification credit for a product with usage history but which is not initially compliant
with these standards. It also applies to modified products for which standard process activities would not be the best option and as
part of the selection process for a new development. This research is applicable to equipment, hardware, and software items.

This research proposes two approaches: bottom-up and top-down. The bottom-up approach furthers the current guidance of
software and hardware levels using qualitative and quantitative criteria for determining the suitability of the collected data in
service. Multistep question-based decision diagrams are proposed to support the determination by the applicant/the designee of
the applicability of the claimed certification credit. Qualitative criteria such as criticality, level of innovation, complexity, and
impact of the fault on the user are assessed through more detailed sets of questions for which any negative answer should lead to
reconsidering the suitability of the data for the intended objective. The similarity in operating environment is evaluated from
investigating potential qualitative and quantitative differences in usage domain and environmental conditions. The quantitative
criteria focus on obtaining and justifying reliability estimates using reliability modeling, for which the selection of a model that
meets prescribed performance objectives is key. Though the reliability of software and AEH are different, the selection of a
reliability model based on statistical properties is applicable to both domains.

The primary limitation of the bottom-up approach is that it requires a significant amount of detailed information that may not be
available through the data collection process and the proposed decision flows are restrictive on the usability of the data. The
top-down approach investigates service history at a system level and abstracts safety goals and system descriptions into an
Equivalent Design Assurance Level (EDAL) that can be directly linked with a minimum service history period. This EDAL,
together with an estimated level of safety, allows for claiming an equivalent level of safety. The recommended minimum service
history is based on Title 14 Code of Federal Regulations Part 25 statistics, but should be transferable to other parts and is
applicable to all design assurance levels.
17. Key Words 18. Distribution Statement
Software service history, Product experience, Reliability, This document is available to the U.S. public through the
Environment, Data collection, Certification credit, DO-178C, National Technical Information Service (NTIS), Springfield,
DO-254, DO-278A, Design assurance, Equivalent level of Virginia 22161. This document is also available from the
safety, Aircraft software, Airborne electronic hardware. Federal Aviation Administration William J. Hughes Technical
Center at actlibrary.tc.faa.gov.
19. Security Classification (of this report) 20. Security Classification (of this page) 21. No. of Pages 22. Price

Unclassified Unclassified 195

Form DOT F 1700.7 (8-72) Reproduction of completed page authorized
ACKNOWLEDGEMENTS
To Cyril Marchand for the approach to software service history and Guy Berthon and B. Durand
for their review of the findings and proposed recommendations.
To the Aerospace Vehicle Systems Institute Authorization for Expenditure 83 project team for
allowing access to its reliability model. In particular, we wish to thank Charles Kilgore, of the
Federal Aviation Administration, and Lori Bechtold, of The Boeing Company, for the engaging
discussions that were ultimately captured in the recommendations for further research section of
this report.

iii
TABLE OF CONTENTS
Page
EXECUTIVE SUMMARY xiii
1. INTRODUCTION 1
1.1 Purpose 1
1.2 Scope 1
1.3 Background 1
1.4 Related Activities and Documents 2
2. APPROACH 3
2.1 Question-based Approach for Applicant and Authority 4
2.2 Bottom-up Approach: Qualitative and Quantitative Criteria 4
2.3 Top-down Approach: Black-box System and Equivalent Level of Safety 4
3. BACKGROUND ON COLLECTING DATA 5
3.1 Data Collection Process 5
3.1.1 Types of Collected Data 6
3.1.2 Failure Reporting 7
3.2 Classification of Systems and Maintenance Actions 8
3.2.1 Non-Repairable and Repairable Systems 8
3.2.2 Types of Maintenance Actions and Impact on Data 10
3.3 Characterization of Faults 12
3.3.1 Consequences of Faults 13
3.3.2 Root Causes of Faults 14
3.3.3 Consideration of Software Architecture and Functions 17
3.4 Specific Difficulties with Data Captured In Service 18
3.4.1 Typical Data from Service Experience 18
3.4.2 Difficulties 19
4. FINDINGS FOR ACCEPTING DATA: QUALITATIVE CRITERIA 21
4.1 Intrinsic Qualitative Aspects 21
4.1.1 Criticality 21
iv
4.1.2 Level of Innovation 22
4.1.3 Complexity 23
4.1.4 Impact on the User 24

4.2 Qualitative Aspects Related to the Environment 24

4.2.1 Facets of Operating Environment 25
4.2.2 Assessment of Usage Domain Characteristics 26
4.2.3 Environmental Conditions 34

4.3 Considering Available Data from Processes 35

4.3.1 Design/Development Process 35
4.3.2 Manufacturing Process 36
4.3.3 Component Process 36
4.3.4 Installation Process 36

4.4 Product Open Problem Reports and Problem-Reporting Systems 37

5. FINDINGS FOR ACCEPTING DATA: QUANTITATIVE CRITERIA 37
5.1 Quantitative Aspects of Reliability Measurement 38

5.1.1 Input Parameters for Reliability Assessment 38
5.1.2 Reliability 40
5.1.3 Difference Between Hardware and Software Reliability 42
5.1.4 Software Reliability in the Design Phase 44

5.2 Reliability Modeling 45

5.2.1 Investigating the Hazard Function 45
5.2.2 Categorizing Models 49
5.2.3 Metrics for Assessing the Model’s Quality of Fit 51

5.3 Approach to Model Selection 52

5.3.1 General Comments on Lifetime Data 52
5.3.2 Selection of Software Reliability Model 54
5.3.3 Fit Objectives and Usability as a Function of DAL 57

5.4 Examples 58

5.4.1 Catalectic Failures Against Technology Scaling 58
5.4.2 Predicting Reliability Growth in Systems 60
5.4.3 Use of Reliability Model and Statistical Properties 62

v
6. FINDINGS FOR SYSTEM-LEVEL BLACK BOX APPROACH 66
6.1 Definitions 66

6.1.1 Area of Use 66
6.1.2 Compliance 67
6.1.3 Description 67
6.1.4 Knowledge Domain 67
6.1.5 Variety 67

6.2 Feedback Principle and Safety Assurance Standards Model 67

6.2.1 Concept of Abstraction 68
6.2.2 Control Abstraction and Information Abstraction 69
6.2.3 Intended Function of the Safety Assurance Standards 70

6.3 System Model, Perception, and EDAL 73

6.3.1 System Model and Perception Level 73
6.3.2 System Model and Equivalent Level of Design Assurance 75

6.4 Determining ELS and ELOS 77

6.4.1 The ELS 77
6.4.2 The ELOS 78

6.5 Assessment of Sufficiency of Service History/Product Experience 79
6.6 Considerations Regarding Problem Reporting 82
6.7 Information to be Provided in Supporting Documentation 83

6.7.1 Estimated Product Part Typology and Complexity 83
6.7.2 Complexity and Software-Based Adaptive Systems 83
6.7.3 Concept of Simplexity 84
6.7.4 Complexity and Statistical Approach 84

7. RECOMMENDATIONS, CONCLUSIONS, AND OPEN DISCUSSION ITEMS 84
7.1 Summary of Findings 84

7.1.1 Data Collection 84
7.1.2 Approaches to Determining Suitability 85
7.1.3 Applicability or Usability 85
7.1.4 Researchers’ Viewpoint 85

7.2 Recommendations for Bottom-Up Approach 86

7.2.1 Determining Suitability of Product Service Experience Data 86
vi
7.2.2 Assessing Similarity of Environment 90
7.2.3 Reliability Modeling 93

7.3 Recommendations for EDAL/ELS Top-down Approach 96

7.3.1 Design Assurance Credit and Product Experience Period 96
7.3.2 Problem Reporting 97
7.3.3 Supporting Documentation 97

7.4 Conclusion 97

7.4.1 Data Collection 97
7.4.2 Using Qualitative and Quantitative Criteria for Software and AEH 98
7.4.3 Using EDAL and ELS at System Level 98
7.4.4 Combined Decision Table 99

7.5 Open Discussion Items 100

7.5.1 Fault Classification Scheme 100
7.5.2 Failure Rate for an Electronic Product 100
7.5.3 Modeling and Quality of Fit Considerations 101
7.5.4 Methods to Improve Software Reliability 102
7.5.5 Comparison with FAA Handbook on Selection of Microprocessors 103

8. GLOSSARY 104

9. REFERENCES 113
APPENDIX A—COMPARISON OF STANDARDS ON SERVICE HISTORY
APPENDIX B—STATISTICS OF LIFETIME DATA
APPENDIX C—RELIABILITY
APPENDIX D—EXAMPLE CONTENT OF IN-SERVICE REPORT
vii
LIST OF FIGURES
Figure Page
1 Sketch of failure-truncated data 6
2 Sketch of time-truncated data 7
3 Non-repairable system lifetime sketch 8
4 Repairable system lifetime sketch 9
5 Failure rate and failure intensity for perfect maintenance 11
6 Failure rate and failure intensity for minimal maintenance 11
7 Failure rate and failure intensity for imperfect maintenance 12
8 Example software architecture illustrating data flow and control flow between layers 17
9 Example of connected LRU or hardware item 28
10 Example of connected LRU or hardware item in target environment 28
11 Identification of differences in external interfaces 29
12 Example of functional configuration for an LRU or hardware item 30
13 Example of functional configuration differences in target environment 31
14 Illustration of data exchange characteristics differences based on frequency 33
15 Illustration of data exchange characteristics differences based on range 33
16 Bathtub curve representation 41
17 Basic shapes of the hazard function 46
18 Typical failure rate observed on classical technologies 58
19 Issues with model validity arise when the wear-out life creeps into the in-service
lifetime portion 59
20 Evolution of the bathtub curve with technology 60
21 Effect of λ and β parameters on reliability growth model 61
22 Cumulative failures against cumulative test time 62
23 Observed system failure rate 63
24 Observed and estimated system failure rates 64
25 Evolution in time of estimator bias 65
26 Evolution in time of estimator standard deviation 66
27 Feedback principle and safety assurance standard model 68
28 Control abstraction representation 69
29 Information abstraction layer 70
viii
30 MBIA of DO-178C software development process 71
31 PL 1 74
32 PL 2 74
33 PL 3 75
34 Proposed graduation shift between F-I DAL and EDAL 81
35 Articulation of EDAL and PL verification to meet F-I DAL 81
36 Decision flow for assessing suitability 88
37 Structured analysis process for evaluating operating environment 91
38 Decision flow for statistical reliability model selection 95
ix