Behaviors of Policy Analysts in Public Investment Decisions: How Policy Analysts Make Judgments

263 Pages·2006·1.73 MB·English

BEHAVIORS OF POLICY ANALYSTS IN PUBLIC INVESTMENT DECISIONS: HOW
POLICY ANALYSTS MAKE JUDGMENTS

by

Kilkon Ko

BS, Yonsei University, 1997

MA, Seoul National University, 2000

Submitted to the Graduate Faculty of

Graduate School of Public and International Affairs in partial fulfillment

of the requirements for the degree of

Doctor of Philosophy

University of Pittsburgh

2006
UNIVERSITY OF PITTSBURGH

FACULTY OF GRADUATE SCHOOL OF PUBLIC AND INTERNATIONAL AFFAIRS

This dissertation was presented

by

Kilkon Ko

It was defended on

March 27, 2006

and approved by

John Mendeloff, PhD

Stephen C. Farber, PhD

Sebastian M. Saiegh, PhD

Aaron M. Swoboda, PhD

Dissertation Director: John Mendeloff, PhD

ii
Copyright by Kilkon Ko
2006
iii
BEHAVIORS OF POLICY ANALYSTS IN PUBLIC INVESTMENT DECISIONS: HOW
POLICY ANALYSTS MAKE JUDGMENTS

Kilkon Ko, PhD

University of Pittsburgh, 2006

Policy analysis emphasizes analytical methods to get better information. Better information,
however, plays a limited role in improving the quality of policy making if it is not appropriately
interpreted. Analytical information measures the different aspects of a policy problem using
different methods: analyses result in information that is created different forms with varying
qualities and relative importance. In order to be more appropriate for policy making, analysts
have to integrate and to interpret the information using contextual and expert knowledge.
However, few studies have paid attention to analysts’ judgment behaviors.
This study examined the judgment structures of analysts who perform actual investment
analysis. I analyzed why politicians and bureaucrats rely on policy analysis, and how a growing
demand for policy analysis leads to an increase in analysts being more actively involved in
investment decision process. Especially, I note that it is not realistic or desirable to restrict the
role of policy analysts as a technical information provider. As analysts are required to consider
multi-dimensional aspects of investment problems, they have to do more integrative analysis
with high level of judgment to respond to the needs of their clients.
Analyses of policy analysts’ judgments show that policy analysts are not obsessed with
economic efficiency when evaluating investment projects. The analysts gave a similar weight to
economic efficiency (51%) as they did to policy factors. Also, the large variation of judgments in
weighting and scoring that was observed can be explained by several factors: the project fields,
iv
analysts’ role in analysis, and their affiliation. Most importantly, we can find strong evidence
that analysts’ judgments are highly related to their self-interests. I showed that analysts’ self-
interests are more problematic in the judgments than the cost underestimation. With the
judgment analyses, I suggest developing management techniques using the statistical distribution,
which allows us to infer the possible range of variation of weighting and scoring.

v
TABLE OF CONTENTS

PREFACE.....................................................................................................................................XI
1.0 INTRODUCTION..............................................................................................................1
1.1 BACKGROUND............................................................................................................1
1.2 SIGNIFICANCE OF RESEARCH.................................................................................8
2.0 LITERATURE REVIEW AND RESEARCH QUESTIONS...........................................13
2.1 POLICY ANALYSIS AND PUBLIC INVESTMENT................................................15
2.1.1 Politicians and Policy Analysis.............................................................................16
2.1.2 Bureaucrats and Policy Analysis..........................................................................21
2.1.3 Policy Analysts in Public Investment...................................................................26
2.2 POLITICS AND THE IMPACT OF ANALYSTS’ JUDGMENTS IN KOREA........32
2.2.1 Accuracy of Analysis............................................................................................33
2.2.2 Ministry of Plan and Budgeting and Pre-Feasibility Study.................................36
2.2.3 Responses from the National Assembly and the Oversight Agency....................39
2.2.4 Potential Problems of the Pre-feasibility Study....................................................43
2.3 HOW POLICY ANALYSTS MAKE JUDGMENTS?................................................45
3.0 RESEARCH DESIGNS....................................................................................................54
3.1 RESEARCH SCOPE....................................................................................................54
3.2 RESEARCH HYPOTHESES.......................................................................................55
3.2.1 Descriptive Analyses............................................................................................56
3.2.2 Test of Self-Interest Hypothesis............................................................................60
4.0 EXPLANATION OF DATA............................................................................................66
4.1 ANALYTIC HIERARCHY PROCESS AND DECISION-MAKING........................67
4.1.1 Background of KDI’s AHP model........................................................................67
4.1.2 Theoretical Background of AHP..........................................................................68
4.1.3 Example of AHP Application...............................................................................73
4.2 CHARACTERISTICS OF ANALYSTS IN KDI AHP DECISION............................81
4.3 DATABASE CONSTRUCTION.................................................................................84
5.0 DESCRIPTIVE ANALYSES: VARIATION OF ANALYSTS’ JUDGMENT...............87
5.1 DESCRIPTIVE ANALYSES OF WEIGHTING ECONOMIC EFFICIENCY...........88
5.1.1 Weighting of Economic Efficiency and Level of Specification...........................90
5.1.2 Weighting of Economic Efficiency and Analysts’ Role in Analysis....................96
5.1.3 Weighting on Economic Efficiency and Affiliation of Analysts..........................98
5.1.4 Linear Model of Economic Efficiency Weighting..............................................101
5.2 DESCRIPTIVE ANALYSIS OF WEIGHTING ON POLICY FACTORS...............104
5.3 WEIGHTS AMONG BASIC POLICY FACTORS...................................................110
5.4 DESCRIPTIVE ANALYSIS OF SCORING..............................................................115
5.5 SUMMARY OF FINDINGS AND IMPLICATIONS...............................................123
6.0 SELF-INTERESTS AND ANALYSTS’ JUDGMENTS...............................................127
6.1 INTENTIONAL BIAS IN COST ESTIMATION......................................................128
vi
6.1.1 The problem of cost underestimation.................................................................128
6.1.2 Research Hypotheses..........................................................................................133
6.1.3 Analyses and Results..........................................................................................136
6.2 INTENTIONAL SCORING.......................................................................................145
6.2.1 Optimistic or Pessimistic Scoring.......................................................................147
6.2.2 Degree of bias and Probability of Acceptance....................................................151
6.2.2.1 The Degree of Bias in Scoring and Project Acceptability (Economic Efficiency)
151
6.2.2.2 The Degree of Bias in Scoring and Project Acceptability (Other Criteria)....154
6.3 ASSOCIATION BETWEEN WEIGHTING AND SCORING..................................161
6.4 RELIABILITY OF JUDGMENT...............................................................................167
6.4.1 Reliability of Weighting.....................................................................................168
6.4.2 Reliability of Scoring..........................................................................................169
6.5 SUMMARY OF FINDINGS AND IMPLICATIONS...............................................171
7.0 STRUCTURE OF JUDGMENT AND ANALYST GROUPS......................................175
7.1 DATA AND METHODS...........................................................................................176
7.2 ROAD PROJECTS.....................................................................................................179
7.2.1 Rail projects........................................................................................................184
7.2.2 “Others” Projects................................................................................................189
7.2.3 Dam and Harbor projects....................................................................................192
7.3 SUMMARY OF FINDINGS AND DISCUSSION....................................................194
8.0 CONCLUSIONS.............................................................................................................196
APPENDIX A.............................................................................................................................204
APPORTIONMENT FORMULAS OF FEDERAL-AID HIGHWAY PROGRAM (TEA-21)
.................................................................................................................................................204
APPENDIX B.............................................................................................................................205
THE CHRONOLOGY OF THE U.S. TRANSPORTATION INVESTMENT.....................205
APPENDIX C.............................................................................................................................211
CRITERIA USED IN PUBLIC INVESTMENT DECISIONS..............................................211
APPENDIX D.............................................................................................................................220
GAME THEORY MODEL UNDER STRATEGIC SITUATION BETWEEN ANALYSTS AND
BUREAUCRATS.......................................................................................................................220
APPENDIX E.............................................................................................................................231
PSYCHOLOGICAL MODEL TOWARD NET BENEFIT AND NET COST..........................231
BIBLIOGRAPHY.......................................................................................................................236

vii
LIST OF TABLES

Table 1 9 Point Scale Used In AHP and Its Semantic Meaning..................................................71
Table 2 Contents and Information Used in Scoring......................................................................75
Table 3 Weighting Results of Four Decision Makers...................................................................77
Table 4 Pairwise comparison matrix of decision maker 1............................................................77
Table 5 Weight Vector of Decision Maker 1................................................................................78
Table 6 Scoring Example of Four Decision Makers.....................................................................79
Table 7 Final Judgment Result of Decision Maker 1...................................................................80
Table 8 Frequency Distribution of Projects by Year and Project Fields......................................84
Table 9 The Number of Analysts by Year, Project Fields, and Affiliation..................................85
Table 10 Weight on the Economic Efficiency..............................................................................92
Table 11 Weight of Economic Factor on Different Levels of Projects by Project Field..............93
Table 12 Result of MANOVA Test..............................................................................................95
Table 13 Weight on Economic Factor by Project Field and Role in Analysis (Specific Project)97
Table 14 Model Fit of Multiple Regression of the Weight on Economic Factor.......................102
Table 15 ANOVA Table of Multiple Regression of the Weight of Economic Factor...............102
Table 16 Estimated Regression Coefficients..............................................................................103
Table 17 Common Factors Used in Public Investment Decisions in KDI Pre-feasibility Studies
.............................................................................................................................................106
Table 18 Project-specific Factors Considered in Pre-Feasibility Study.....................................107
Table 19 Weight on Basic Policy Factors Compared to Project-Specific Factors: All Investment
Decisions.............................................................................................................................108
Table 20 Weight between Basic Policy Factors and Project-specific Policy Factor by Project
Field....................................................................................................................................109
Table 21 Weights among Basic Policy Sub-factors....................................................................111
Table 22 Weights among Basic Policy Sub-factors by Project Field.........................................113
Table 23 Multiple Regression of Basic Policy Factors...............................................................114
Table 24 Multiple Measures for Variation of Scoring................................................................119
Table 25 Average Estimated Project Cost by Project Fields......................................................122
Table 26 Descriptive Statistics and 95% Confidence Limits of Cost Underestimate Rate........139
Table 27 Correlation between the Size of Project and Cost Underestimation............................140
Table 28 Model Fit of Hierarchical Linear Model......................................................................142
Table 29 Estimation of Random Components............................................................................142
Table 30 Coefficient Estimates of Hierarchical Linear Model on Cost Underestimation.........143
Table 31-1 Deviation of Scoring by Project Field, by Affiliation (Economic Efficiency).........148
Table 32 Impact of BC ratio on Optimistic and Pessimistic Bias...............................................153
Table 33 Impact of Underdevelopment Index (UI) on Positive and Negative Scoring..............155
Table 34 Impact of VA_GRDP to Optimistic and Pessimistic Scoring.....................................157
Table 35 Deviation of Judgment and Project Acceptability.......................................................159
Table 36 Inconsistency Ratio by Affiliation and Project Field...................................................169
viii
Table 37 DPF (Deviance index of Policy Feasibility) by Affiliation and Project field..............170
Table 38 Generalized Squared Distance among Analyst Groups (Road Projects).....................180
Table 39 Canonical Discriminant Analysis (Road Projects)......................................................180
Table 40 Coefficients of Canonical Discriminant Functions (Road Projects)............................181
Table 41 The Centroid of Each Group (Road Projects)..............................................................182
Table 42 Univariate R-Square and Discriminant Loading (Road Projects)...............................183
Table 43 Partial R-Square of Stepwise Variable Selection (Road Projects)..............................183
Table 44 Classification Accuracy (Road Projects).....................................................................184
Table 45 Generalized Squared Distance among Analyst Group (Rail Projects)........................185
Table 46 Canonical Discriminant Analysis (Rail Projects)........................................................186
Table 47 Coefficients of Canonical Discriminant Functions (Rail Projects).............................186
Table 48 The Centroid of Each Group (Rail Projects)...............................................................187
Table 49 Univariate R-Square and Discriminant Loading (Rail Projects).................................188
Table 50 Partial R-Square of Stepwise Variable Selection (Rail Projects)................................188
Table 51 Classification Accuracy (Rail Projects).......................................................................189
Table 52 Generalized Squared Distance among Analyst Group (Others Projects)....................190
Table 53 Canonical Discriminant Analysis (Others Projects)....................................................191
Table 54 Coefficients of Canonical Discriminant Functions (Others Projects).........................191
Table 55 Centroid of Each Group (Others Projects)...................................................................192
Table 56 Generalized Squared Distance among Analyst Group (Dam and Harbor Projects)....193
Table 57 Number of Urban Transportation Decision Cases in which Each Factor Was
Considered or Important.....................................................................................................213
Table 58 Factors Used in Transportation Decisions (Considered in Current Literatures).........214
Table 59 Factors Used in Transportation Decisions (Considered by State Government:
Washington State Case)......................................................................................................215
Table 60 Key Factors to be considered in Transportation Investment (Identified by U.S. GAO)
.............................................................................................................................................217
Table 61 Factor Used in Transportation Decisions (the Korean Government Case).................218

ix
LIST OF FIGURES

Figure 1 The Proportion of Korean Legislators Who Were Newly Elected.................................21
Figure 2 Increasing Roles of Policy Analysts in Public Investment.............................................31
Figure 3 Numbers of Articles Referring the Pre-feasibility Study (2000-2005)..........................38
Figure 4 Integrative Policy Analysis Process...............................................................................48
Figure 5 Framework of Analysis..................................................................................................65
Figure 6 Basic Decision Hierarchy of Pre-Feasibility Study........................................................75
Figure 7 Weight on Economic Factor by Project Field and Role in Analysis (General Public
Investment)...........................................................................................................................98
Figure 8 Weight on Economic Factor by Project Field and Affiliation of Analysts..................100
Figure 9 Weight on Basic Policy Factors over Project-specific Factors, by Project Field, by Role
of Analysts..........................................................................................................................109
Figure 10 Estimated Average Standard Deviation of Scoring by Project Fields (Criteria Using
Quantitative Measures).......................................................................................................121
Figure 11 Estimated Average Standard Deviation of Scoring by Project Fields (Criteria Using
Qualitative Measures).........................................................................................................121
Figure 12 Cost Underestimation Rate by Year (2001 to 2004)..................................................138
Figure 13 Deviation of Scoring in Three Different Levels of BC ratios....................................152
Figure 14 Response Lines of Deviation to BC ratio...................................................................154
Figure 15 Response Lines of Deviation of Scoring on BW According to UI............................156
Figure 16 Deviation of Judgment and Project Acceptability (25% and 75% Quartiles as
Threshold Values for Low, Medium and High Project Acceptability)...............................160
Figure 17 Choice of Weighting and Scoring, by Criteria (KDI Analysts).................................164
Figure 18 Choice of Weighting and Scoring, by Criteria (Private Analysts).............................166
Figure 19 Equilibrium under Imperfect Information..................................................................228
Figure 20 Equilibrium under Perfect Information......................................................................229
Figure 21 Hypothetical Patterns of Attitude Toward Net Costs and Benefits in Scoring Project’s
Acceptability.......................................................................................................................234
Figure 22 Estimated Scoring on Economic efficiency of KDI and Private Analysts.................235

x