Program Evaluation: A Federal Agency's Air Traffic Control Train-the-Trainer Program

146 Pages·2015·1.01 MB·English

Walden University
ScholarWorks
Walden Dissertations and Doctoral Studies
Walden Dissertations and Doctoral Studies
Collection
2015
Program Evaluation: A Federal Agency's Air Traffic
Control Train-the-Trainer Program
Lisa Marie Mercer
Walden University
Follow this and additional works at:https://scholarworks.waldenu.edu/dissertations
Part of theAdult and Continuing Education Administration Commons,Adult and Continuing
Education and Teaching Commons,Educational Assessment, Evaluation, and Research Commons,
and theMilitary and Veterans Studies Commons
This Dissertation is brought to you for free and open access by the Walden Dissertations and Doctoral Studies Collection at ScholarWorks. It has been
accepted for inclusion in Walden Dissertations and Doctoral Studies by an authorized administrator of ScholarWorks. For more information, please
[email protected].
Walden University

COLLEGE OF EDUCATION

This is to certify that the doctoral study by

Lisa Mercer

has been found to be complete and satisfactory in all respects,
and that any and all revisions required by
the review committee have been made.

Review Committee
Dr. Debra Beebe, Committee Chairperson, Education Faculty
Dr. Jean Sorrell, Committee Member, Education Faculty
Dr. Karen Hunt, University Reviewer, Education Faculty

Chief Academic Officer

Eric Riedel, Ph.D.

Walden University
2015
Abstract
Program Evaluation: A Federal Agency’s Air Traffic Control Train-the-Trainer Program
by
Lisa M. Mercer

MAS, Embry-Riddle Aeronautical University (ERAU), 2011
BSA, ERAU, 2005
Instructional System Development Graduate Certification, ERAU, 2013

Doctoral Study Submitted in Partial Fulfillment
of the Requirements for the Degree of
Doctor of Education

Walden University
December 2015
Abstract
In 2014, the Federal Aviation Administration (FAA) highlighted to the U.S. Senate the
need to focus on air traffic control (ATC) training to meet job qualification and attrition
rates within the career field. One U.S. Department of Defense military service assists the
FAA in providing worldwide ATC services. This service is referred to as the agency
throughout this paper to ensure confidentiality. The agency’s ATC career field manager
echoed the FAA’s call for action in his 2014 Strategic/Action Plan. In August 2013, the
agency’s ATC trainer program was published. As of December 2015, the program had
not been evaluated. The purpose of this study was to ascertain if the program facilitated
the learning of critical ATC on-the-job training skills. An ad hoc expertise-oriented
evaluation was conducted using the lenses of andragogy, experiential learning, and
instructional system design (ISD). Purposeful sampling procedures were used to select
20 participants across the subgroups of supervisors, trainers, managers, and training
developers from 7 focus sites. The semi-structured interviews queried 4 topical areas
derived from Kirkpatrick’s 4 levels of evaluation model. Data collected via documents
and interviews were analyzed using descriptive, emotion, eclectic, and pattern coding.
Key findings indicated that the program was not developed compliant with ISD principles
and did not promote adult learning as endorsed by andragogy and experiential learning
theory. The implications for positive social change include providing stakeholders with
data needed to make evidence-based decisions regarding the current and future state of
the program. The evaluation report project can be shared with the FAA, an agency
partner, and has the potential to create a platform for improved training practices focusing
on optimum and successful adult learning transactions.
Program Evaluation: A Federal Agency’s Air Traffic Control Train-the-Trainer Program
by
Lisa M. Mercer

MAS, Embry-Riddle Aeronautical University (ERAU), 2011
BSA, ERAU, 2005
Instructional System Development Graduate Certification, ERAU, 2013

Doctoral Study Submitted in Partial Fulfillment
of the Requirements for the Degree of
Doctor of Education

Walden University
December 2015
Dedication
I dedicate this work to all past, present, and future agency air traffic controllers.
Agency controllers provide service to aircraft in the global air traffic system during both
times of peace and times of war. Air traffic controllers are highly specialized and
routinely make split-second decisions that ensure the continued safety of millions of
dollars in assets and human life. Air traffic controllers are meticulous and assertive
decision makers who possess excellent real-time risk analysis skills and remain calm
under extreme pressure. Moreover, agency controllers are a cohesive team that embodies
the teamwork ethos both on and off duty. Once an agency controller, you are a teammate
for life. Thank you to all my teammates for what you selflessly do every day for each
other and the skies above without regard to personal gain or glory. In particular, thank
you to the 20 agency controllers who shared their perspectives, experiences, and
recommendations for this study.
Acknowledgments
To my biggest supporter, Michael, thank you for all the sacrifices you made in
support of my doctoral dream. You are the rock to my storm, and without you, this
would not have been possible. To our children, Raistlin and Regin, I am so proud of you
little ones! May you aspire to always reach for the stars, always land on your feet, and
always know I love you to Pluto and back a trillion times.
To my parents, who never lost faith when I veered off course (more than once),
thank you for having faith that I would find my way (eventually). Thank you for
teaching me to value hard work and to never settle for mediocrity. To my grandparents, I
will be forever grateful for all you shared about life, love, and God. To my sisters, you
are amazing women who lift my spirits daily and inspire me to achieve great things by
simply following your incredible example.
Felisha, dear friend, I am happy we traveled this road together. It is time for a
Gone with the Wind marathon! Sherry Jones and Colleen Geier, without your quirky
jokes, I would have surely gone mad. Thank you to all the amazing agency supervisors I
have had over the years. Trish Fisher, my first supervisor, thank you for encouraging me
to take my first academic step all those years ago. Thank you to all my professors and
dissertation committee members. Special thanks to Dr. Debra Beebe, my doctoral chair,
who encourages all her M & M’s to never to give up even when the fog is thick and the
shoreline is out of sight.
Table of Contents
List of Tables ...................................................................................................................... iv

List of Figures ...................................................................................................................... v

Section 1: The Problem ....................................................................................................... 1

Introduction ......................................................................................................................... 1

Definition of the Problem .............................................................................................. 2

Rationale ........................................................................................................................ 3

Evidence of the Problem at the Local Level ........................................................... 3

Evidence of the Problem From the Professional Literature .................................... 5

Definitions ..................................................................................................................... 5

Significance ................................................................................................................... 6

Research Focus .............................................................................................................. 7

Review of the Literature ................................................................................................ 7

Theoretical/Conceptual Frameworks ................................................................. 8

Implications ................................................................................................................. 13

Summary ...................................................................................................................... 13

Section 2: The Methodology ............................................................................................. 15

Introduction ................................................................................................................. 15

Research Design and Approach ................................................................................... 15

Participants .................................................................................................................. 17

Protection of Human Subjects ......................................................................... 20

Data Collection ............................................................................................................ 22

Potential Bias ......................................................................................................... 26

i
Data Analysis and Findings ......................................................................................... 27

Limitations ................................................................................................................... 66

Conclusion ................................................................................................................... 67

Section 3: The Project ....................................................................................................... 69

Introduction ................................................................................................................. 69

Description and Goals ................................................................................................. 69

Rationale ...................................................................................................................... 70

Review of the Literature .............................................................................................. 73

Implementation ............................................................................................................ 78

Project Evaluation ....................................................................................................... 79

Implications Including Social Change ......................................................................... 80

Local Community .................................................................................................. 80

Far-Reaching ......................................................................................................... 81

Conclusion ................................................................................................................... 82

Section 4: Reflections and Conclusions ............................................................................ 83

Introduction ................................................................................................................. 83

Project Strengths .......................................................................................................... 83

Limitations ................................................................................................................... 84

Recommendations for Alternative Approaches ........................................................... 85

Scholarship .................................................................................................................. 86

Project Development and Evaluation .......................................................................... 87

Leadership and Change ............................................................................................... 87

Analysis of Self as Scholar .......................................................................................... 88

ii
Analysis of Self as Practitioner ................................................................................... 88

Analysis of Self as Project Developer ......................................................................... 90

The Project’s Potential Impact on Social Change ....................................................... 90

Implications, Applications, and Directions for Future Research ................................ 91

Conclusion ................................................................................................................... 91

References ......................................................................................................................... 93

Appendix A: Program Evaluation Report ....................................................................... 103

Appendix B: Document Review Instrument ................................................................... 133

Appendix C: Interview Protocol ...................................................................................... 134

iii