An Approximate Dynamic Programming Approach to Financial Execution for Weapon System ...

168 Pages·2013·1.48 MB·English

by  Erich D. Morman

Approved for Public Release
13-MDA-7231 (5 April 13)

AN APPROXIMATE DYNAMIC PROGRAMMING APPROACH TO
FINANCIAL EXECUTION FOR WEAPON SYSTEM PROGRAMS

by

Erich D. Morman
A Dissertation
Submitted to the
Graduate Faculty
of
George Mason University
in Partial Fulfillment of
The Requirements for the Degree
of
Doctor of Philosophy
Systems Engineering and Operations Research

Dr. Rajesh Ganesan, Dissertation Director
Dr. Karla Hoffman, Committee Member
Dr. Andrew Loerch, Committee Member
Dr. Mark Pullen, Committee Member
Dr. Ariela Sofer, Department Chair
Dr. Kenneth S. Ball, Dean

Volgenau School of Engineering

Spring Semester 2013
Date: 5 April 2013
George Mason University

Fairfax, VA
Approved for Public Release
13-MDA-7231 (5 April 13)
An Approximate Dynamic Programming Approach to Financial Execution for Weapon
System Programs
A dissertation submitted in partial fulfillment of the requirements for the degree of
Doctor of Philosophy at George Mason University
by
Erich D. Morman
Master of Science
George Mason University, 2002
Bachelor of Arts
University of Dayton, 1996
Director: Rajesh Ganesan, Professor
Department of Systems Engineering and Operations Research
Spring Semester 2013
George Mason University
Fairfax, VA
This work is licensed under a creative commons
attribution-noderivs 3.0 unported license.

ii
DEDICATION
For my mom, dad, and sister.
iii
ACKNOWLEDGEMENTS
I owe a tremendous amount of thanks and gratitude to all those who have assisted me
over the many years it has taken to complete this research. The ability to obtain and
complete a doctorate education was only possible due to the continual efforts,
encouragement, and wisdom provided by the faculty and staff at George Mason
University, my fellow colleagues at the Missile Defense Agency (MDA) as well as my
family and friends.
I’d like to especially acknowledge Dr. Rajesh Ganesan for serving as my advisor.
His accessibility and willingness to spend countless hours discussing mathematical
theories, modeling strategies, and interpreting results were critical to maintaining
momentum on this project. Additionally, his contagious enthusiasm for the field of
Approximate Dynamic Programming (ADP) was instrumental in keeping me motivated
and challenged. It was my pleasure to have worked with you on this endeavor.
I am also thankful to have had Dr. Karla Hoffman, Dr. Andrew Loerch, and Dr.
Mark Pullen serve on my committee. I am grateful for Dr. Hoffman’s insightful feedback
and review of both my proposal and dissertation materials. Her knowledge on financial
or resource allocation problems helped me appreciate the finer points of modeling
structure and implementation. Dr. Loerch’s perspective on weapon system modeling,
military analysis, as well as client-customer relationships was extremely beneficial and
very helpful. I’d also like to thank Dr. Pullen for his willingness to be on the committee
and providing me perhaps the best advice which was that the secret to completing a Ph.D.
is not necessarily intelligence but, persistence. Lastly, I’d like to thank Angel Manzo,
Josefine Wiecks, Sally Evans, and Lisa Nolder for their professionalism and help in
accomplishing all the necessary administrative tasks required to complete this degree.
I’d like to thank my many clients, colleagues, and friends associated with MDA
who supported this project. I owe a special thanks to Ms. Jan Bond whose early on
support and encouragement was absolutely critical to removing the roadblocks that
helped initiate this research. Also, I’d like to acknowledge Mr. Terry Little for his
willingness to review and discuss my draft materials. Lastly, I’d like to recognize Mr.
Richard Ritter, Jim Watzin, Rob Cramer, Phil Gretzkowski, Scott Hillstead, David
Lancaster, Sue Knapp, Alex Snead, Rob Cramer, Chris Szkrybalo, and Tammy Sledge
for their support at all levels in helping me obtain the resources, perform the
coordination, and navigate the administrative requirements necessary to accomplish this
work at MDA.

iv
TABLE OF CONTENTS
Page
List of Tables .................................................................................................................... vii
List of Figures .................................................................................................................. viii
List of Equations ................................................................................................................. x
List of Abbreviations and Symbols................................................................................... xii
Abstract ............................................................................................................................. xii
Chapter One – The Problem ............................................................................................... 1
1.1 Problem Statement .................................................................................................... 1
1.2 Execution Definitions ................................................................................................ 3
1.3 State-Space Vector .................................................................................................... 4
1.4 Problem Environment ............................................................................................... 5
1.5 Markov Decision Process .......................................................................................... 8
1.6 Dynamic Programming (DP) Introduction .............................................................. 11
1.7 Allocation Parameter ............................................................................................... 16
1.8 Why Use Approximate Dynamic Programming (ADP) ......................................... 17
1.9 Contributions and Structure of Dissertation ............................................................ 19
Chapter Two – ADP Concepts and Literature Overview ................................................. 21
2.1 Dynamic Programming Background ....................................................................... 21
2.2 Problems with Dynamic Programming ................................................................... 23
2.3 “Curse of Modeling” ............................................................................................... 24
2.4 “Curse of Dimensionality” ...................................................................................... 25
2.5 The Need for Approximate Dynamic Programming (ADP) ................................... 26
2.6 Transition Function ................................................................................................. 27
2.7 Bellman’s Equation ................................................................................................. 30
2.8 Sampling the Value Function .................................................................................. 32
2.9 Value Function Update............................................................................................ 33
2.10 Q-Learning on the Pre-Decision State .................................................................. 33
v
2.11 Arguments for Using the Post Decision State (PDS) ............................................ 36
2.12 Q-Learning on the Post Decision State (PDS) ...................................................... 39
2.13 Value Function Learning Algorithm ..................................................................... 41
2.14 Q-Learning and Value Function Learning Design Summary ............................... 43
Chapter Three – The Model Design.................................................................................. 44
3.1 Perspectives on Data Structures .............................................................................. 44
3.2 Immediate Cost Function and the ADP Network.................................................... 47
3.3 Subroutines of the ADP Model ............................................................................... 51
3.4 Complexities Due to Adding Multiple Projects ...................................................... 58
3.5 Q-Learning and Value Function Learning Designs ................................................ 67
3.6 Convergence: Alpha-Decay .................................................................................... 71
3.7 Convergence: Mean Square Error (MSE) ............................................................... 75
3.8 Exploration Vs. Exploitation (Learning) ................................................................. 77
Chapter Four – Results and Analysis ................................................................................ 80
4.1 Learnt Phase ............................................................................................................ 80
4.2 Comparative Results ............................................................................................... 83
4.3 Collected Data ......................................................................................................... 85
4.4 Model Input Examples ............................................................................................ 87
4.5 Model Output Examples.......................................................................................... 89
4.6 Learnt Phase Observations from Test Case #2 – Trial #1 ....................................... 94
4.7 Exploration Vs. Exploitation (Learning) Revisited................................................. 96
4.8 Learnt Phase Observations from Test Case #2 – Trial #3 ....................................... 97
4.9 Learnt Phase Sensitivity Analysis ......................................................................... 101
4.10 Learnt Phase Observations from Test Case #3 – Trial #2 ................................... 103
4.11 Additional Test Cases and Analysis .................................................................... 121
Chapter Five – Conclusions, Contributions, Future Research, and Next Steps .............. 136
5.1 Conclusions ........................................................................................................... 136
5.2 Contributions ......................................................................................................... 137
5.3 Future Research ..................................................................................................... 140
5.4 Next Steps ............................................................................................................. 144
References ....................................................................................................................... 149

vi
LIST OF TABLES
Table Page
Table 1: OSD Benchmarks for Obligations and Expenditures ........................................... 2
Table 2: Investment Three NPV ....................................................................................... 14
Table 3: Investment One and Two NPV ........................................................................... 15
Table 4: Multiple Projects Allocation Parameter .............................................................. 61
Table 5: Example Viable Commitment Allocations for Multiple Projects ....................... 66
Table 6: Test Case Summaries ........................................................................................ 122
Table 7: Test Case Summaries (Cont.) ........................................................................... 123

vii
LIST OF FIGURES
Figure Page
Figure 1: Example Obligation and Expenditure End-Month April Status .......................... 7
Figure 2: Backward Recursion Network........................................................................... 13
Figure 3: The Need for ADP ............................................................................................. 27
Figure 4: Transition Function Timeline ............................................................................ 29
Figure 5: Funding Planning Matrices ............................................................................... 45
Figure 6: Initial Planning $5.0M Project .......................................................................... 49
Figure 7: ADP Network Model of Commitment Cash Flow Problem ............................. 50
Figure 8: Matrix A & B Commitment Action Selection................................................... 52
Figure 9: Matrix A & B Exogenous Updates.................................................................... 56
Figure 10: Matrix A & B Planning Updates ..................................................................... 57
Figure 11: Updated Matrix A & B Planning Data for Multiple Projects .......................... 60
Figure 12: Multiple Projects Matrix B Data ..................................................................... 62
Figure 13: Q-Learning Diagram ....................................................................................... 67
Figure 14: Example Q-Matrix ........................................................................................... 69
Figure 15: Value Function Learning Diagram and Output Vector ................................... 70
Figure 16: PDS Value Function Estimates Convergence Patterns ................................... 76
Figure 17: Mean Square Error (MSE) Plot ....................................................................... 77
Figure 18: Alpha-Decay for Exploration and Exploitation............................................... 79
Figure 19: Learnt Phase Inputs ......................................................................................... 82
Figure 20: Test Case #2 – Trial #1 Input Parameters ....................................................... 88
Figure 21: Test Case #2 – Trial #1 Matlab Structure........................................................ 89
Figure 22: Test Case #2 – Trial #1 Mean Square Error (MSE) ........................................ 90
Figure 23: Test Case #2 – Trial #1 Output Graphic 1 ....................................................... 92
Figure 24: Test Case #2 – Trial #1 Output Graphic 2 ....................................................... 93
Figure 25: Test Case #2 – Trial #3 Input Parameters and MSE ....................................... 98
Figure 26: Test Case #2 – Trial #3 Output Graphic 1 ....................................................... 99
Figure 27: Test Case #2 – Trial #3 Output Graphic 2 ..................................................... 100
Figure 28: Learnt Phase Matlab Inputs ........................................................................... 103
Figure 29: Test Case #3 – Trial #2 Input Parameters and MSE ..................................... 105
Figure 30: Test Case #3 – Trial #2 Output Graphic 1 / ‘exog’ = All .............................. 106
Figure 31: Test Case #3 – Trial #2 Output Graphic 2 / ‘exog’ = All .............................. 107
Figure 32: Test Case #3 – Trial #2 Output Graphic 1 / ‘exog’ = High .......................... 108
Figure 33: Test Case #3 – Trial #2 Output Graphic 2 / ‘exog’ = High .......................... 109
Figure 34: Test Case #3 – Trial #2 Output Graphic 1 / ‘exog’ = Low ........................... 110
Figure 35: Test Case #3 – Trial #2 Output Graphic 2 / ‘exog’ = Low ........................... 111
viii
Figure 36: Test Case #3 – Trial #2 Output Graphic 1 / ‘exog’ = Low to High ............. 112
Figure 37: Test Case #3 – Trial #2 Output Graphic 2 / ‘exog’ = Low to High ............. 113
Figure 38: Test Case #3 – Trial #2 Output Graphic 1 / ‘exog’ = High to Low ............. 114
Figure 39: Test Case #3 – Trial #2 Output Graphic 2 / ‘exog’ = High to Low ............. 115
Figure 40: End of Fiscal Year Commitment Amounts ................................................... 119
Figure 41: End of Month May Commitment Levels ...................................................... 120
Figure 42: Test Case #3 – Trial #2 MSE and Variance Graphics ................................... 126
Figure 43: Test Case #3 – Trial #3 MSE and Variance Graphics ................................... 127
Figure 44: Test Case #3 – Trial #5 Alpha-Decay, MSE, and Variance Graphics .......... 130
Figure 45: Test Case #3 – Trial #6 MSE and Variance Graphics ................................... 131
Figure 46: Test Case #4 – Trial #1 Input Parameters and MSE ..................................... 132
Figure 47: Test Case #4 – Trial #1 Output Graphic 2 / ‘exog’ = All .............................. 133
Figure 48: Test Case #4 – Trial #1 Output Graphic 2 / ‘exog’ = High ........................... 134
Figure 49: Test Case #4 – Trial #1 Output Graphic 2 / ‘exog’ = Low ........................... 135

ix