Identifying the limits of an integrated training environment using human abilities and affordance

356 Pages·2013·4.13 MB·English

by  Hodges, Glenn August

NAVAL
POSTGRADUATE
SCHOOL

MONTEREY, CALIFORNIA

DISSERTATION

IDENTIFYING THE LIMITS OF AN
INTEGRATED TRAINING ENVIRONMENT
USING HUMAN ABILITIES AND AFFORDANCE
THEORY

by

Glenn August Hodges

June 2014

Dissertation Supervisor: Rudolph Darken
This dissertation was performed at the MOVES Institute
Approved for public release; distribution is unlimited
THIS PAGE INTENTIONALLY LEFT BLANK
REPORT DOCUMENTATION PAGE
Form Approved OMB No. 0704-0188
Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for
reviewing instruction, searching existing data sources, gathering and maintaining the data needed, and completing and
reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of
information, including suggestions for reducing this burden, to Washington headquarters Services, Directorate for Information
Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302, and to the Office of
Management and Budget, Paperwork Reduction Project (0704-0188) Washington DC 20503.
1.AGENCY USE ONLY (Leave blank) 2.REPORT DATE 3.REPORT TYPE AND DATES COVERED
June 2014 Dissertation

4.TITLE AND SUBTITLE IDENTIFYING THE LIMITS OF AN INTEGRATED 5.FUNDING NUMBERS
TRAINING ENVIRONMENT USING HUMAN ABILITIES AND AFFORDANCE
THEORY
6.AUTHOR Glenn August Hodges
7.PERFORMING ORGANIZATION NAME AND ADDRESS 8.PERFORMING
Naval Postgraduate School ORGANIZATION REPORT
Monterey, CA 93943-5000 NUMBER
9.SPONSORING / MONITORING AGENCY NAME AND ADDRESS 10.SPONSORING / MONITORING
Executive Agent for Simulations to Mission Command Interoperability (SIMCI), AGENCY REPORT NUMBER
PEO STRI / PEO C3T, 12350 Research Parkway
Orlando, FL 32826-3276
11. SUPPLEMENTARY NOTES The views expressed in this thesis are those of the author and do not reflect the official
policy or position of the Department of Defense or the U.S. Government. IRB protocol number ___N/A_____________.
12a. DISTRIBUTION / AVAILABILITY STATEMENT 12b. DISTRIBUTION CODE
Approved for public release; distribution is unlimited A
13.ABSTRACT (maximum 200 words)
This research investigated the development and use of an analytical assessment methodology anchored in systems engineering
principles, affordance theory, and human abilities, to measure the potential of integrated training environments (ITEs) to
effectively support training. Empirical investigation of ITEs is costly, lacks formal guidance, and is therefore often unreliable.
Ad hoc studies, commissioned by individual organizations, constitute the current state of Army ITE evaluation. These
assessments are often entirely based on subject matter expert judgment through surveys, which produce results that are linked
indirectly and loosely to the ITEs. What is required is a repeatable, inexpensive, analytical approach to ITE assessment that
bounds the potential of a given system to the support it provides to the deliberate practice of specific tasks. The results of this
research include the development and use of the integrated training environment assessment methodology (ITEAM). ITEAM
was used to evaluate the ability of several ITEs to support the deliberate practice of specific tasks during training. The
dissertation shows that ITEAM consistently predicted where training was supported by an ITE and generally how well. ITEAM
is offered as a tool to be used early in the acquisition process to affordably define and verify the requirements of candidate ITE
solutions for Department of Defense needs.
14. SUBJECT TERMS Integrated training environment, training assessment, human abilities, 15.NUMBER OF
fidelity, affordance theory, task analysis, verification, requirements analysis, simulations, TADSS PAGES
355
16.PRICE CODE
17.SECURITY 18.SECURITY 19.SECURITY 20.LIMITATION
CLASSIFICATION OF CLASSIFICATION OF THIS CLASSIFICATION OF OF ABSTRACT
REPORT PAGE ABSTRACT
Unclassified Unclassified Unclassified UU
NSN 7540-01-280-5500 Standard Form 298 (Rev. 2-89)
Prescribed by ANSI Std. 239-18
i
THIS PAGE INTENTIONALLY LEFT BLANK
i i
Approved for public release; distribution is unlimited
IDENTIFYING THE LIMITS OF AN INTEGRATED TRAINING
ENVIRONMENT USING HUMAN ABILITIES AND AFFORDANCE
THEORY
Glenn August Hodges
Lieutenant Colonel, United States Army
B.S., Old Dominion University, 1993
M.S., Naval Postgraduate School, 2004
Submitted in partial fulfillment of the requirements for the degree of
DOCTOR OF PHILOSOPHY IN
MODELING, VIRTUAL ENVIRONMENTS AND SIMULATION
from the
NAVAL POSTGRADUATE SCHOOL
June 2014
Author: Glenn August Hodges
Approved by: Rudolph Darken Michael McCauley
Professor, Computer Science and Research Professor, Operations
MOVES Research and MOVES
Supervisor and Committee Chair
Joseph Sullivan Eugene Paulo
Director, MOVES Institute Associate Professor, Systems
Engineering and MOVES
Jeffrey Appleget
Senior Lecturer, Operations
Research and MOVES
Approved by: Peter Denning, Chair, Department of Computer Science
Approved by: Douglas Moses, Vice Provost for Academic Affairs
ii i
THIS PAGE INTENTIONALLY LEFT BLANK
iv
ABSTRACT
This research investigated the development and use of an analytical assessment
methodology anchored in systems engineering principles, affordance theory, and human
abilities, to measure the potential of integrated training environments (ITEs) to
effectively support training. Empirical investigation of ITEs is costly, lacks formal
guidance, and is therefore often unreliable. Ad hoc studies, commissioned by individual
organizations, constitute the current state of Army ITE evaluation. These assessments are
often entirely based on subject matter expert judgment through surveys, which produce
results that are linked indirectly and loosely to the ITEs. What is required is a repeatable,
inexpensive, analytical approach to ITE assessment that bounds the potential of a given
system to the support it provides to the deliberate practice of specific tasks. The results of
this research include the development and use of the integrated training environment
assessment methodology (ITEAM). ITEAM was used to evaluate the ability of several
ITEs to support the deliberate practice of specific tasks during training. The dissertation
shows that ITEAM consistently predicted where training was supported by an ITE and
generally how well. ITEAM is offered as a tool to be used early in the acquisition process
to affordably define and verify the requirements of candidate ITE solutions for
Department of Defense needs.

v
THIS PAGE INTENTIONALLY LEFT BLANK
v i
TABLE OF CONTENTS
I. INTRODUCTION .............................................................................................................1
A. A STORY TO SET THE STAGE ..................................................................1
B. INSIGHT INTO THE CULTURE OF ARMY TRAINING ........................2
C. DISCUSSION OF THE PROBLEM ..............................................................3
D. RESEARCH APPROACH ..............................................................................6
E. CLARIFICATION OF TERMS .....................................................................8
F. DOCUMENT ORGANIZATION...................................................................8
II. LITERATURE REVIEW ...............................................................................................11
A. FRONT-END ANALYSIS.............................................................................12
1. Capability Based Assessments ..........................................................13
2. Training Objectives and Performance Requirements ....................15
3. Task Analysis ......................................................................................16
4. Human Ability Requirements ...........................................................18
a. Introduction .............................................................................18
b. Impetus for Development ........................................................19
c. Relationship to This Research ................................................19
B. FIDELITY, AFFORDANCE THEORY AND TRANSFER OF
TRAINING .....................................................................................................20
1. Fidelity ................................................................................................20
2. Affordance Theory .............................................................................22
3. Transfer of Training ..........................................................................24
C. VERIFICATION AND VALIDATION .......................................................28
D. IMPACTS OF SUBJECT MATTER EXPERTS........................................29
1. Positive Contributions .......................................................................30
2. Negative Impacts ................................................................................30
3. Recommendation................................................................................31
E. SIMULATION ASSESSMENT ....................................................................32
1. Non-Training Domains ......................................................................32
2. Within a Training Domain ................................................................34
F. TRAINING EFFECTIVENESS ANALYSIS (TEA) SYSTEM .................39
1. Overview of the TEA System ............................................................39
2. TRADOC Regulation 350-32 ............................................................40
3. User View of the TEA System ...........................................................42
4. Conclusion ..........................................................................................45
G. SUMMARY ....................................................................................................46
III. INTEGRATED TRAINING ENVIRONMENT ASSESSMENT
METHODOLOGY ....................................................................................................49
A. DESCRIPTION OF THE ITEAM PROCESSES .......................................51
1. Requirements Definition ...................................................................51
vi i
a. Describe the Operational Need, Capability or
Performance Shortfall in Terms That are Specific and
Measurable ..............................................................................51
b. Determine and Describe the ITE Capabilities Using an
Operational Concept and Functional Allocation ..................51
c. Conduct a Job/Task Analysis of the Real World Activity to
Identify the Critical Tasks Needed to Accomplish the
Desired Learning/Training Objectives ...................................52
d. Annotate the Job/Task Analysis with Human Ability
Requirements Associated with the Tasks ...............................52
e. Determine and Describe the Necessary ITE Affordance
Requirements Based on the Identified Human Abilities .......52
2. Verification .........................................................................................52
a. Determine What Human Abilities the ITE Supports
Compared to Those Necessary to Execute the Real World
Tasks ........................................................................................53
b. Determine the Affordances Present in the ITE .....................53
3. Assessment ..........................................................................................54
a. Quantify the Level of Support the ITE Provides to the
Training Audience ..................................................................54
B. SUBTASK AFFORDANCE SCORING ......................................................55
1. Subtask Affordances are Unique ......................................................56
2. Subtask Affordances Previously Accounted For ............................58
3. Subtask Affordances Partially Unique.............................................61
a. Step One...................................................................................61
b. Step Two ..................................................................................61
c. Step Three ................................................................................62
4. Subtask Affordances Contained in Multiple Analyses ...................63
C. PROCEDURE FOR TASK SCORING .......................................................64
D. SUMMARY ....................................................................................................64
IV. CASE STUDY I: GAME-BASED TRAINING IN AN OPERATIONAL
SETTING ....................................................................................................................67
A. METHOD .......................................................................................................67
1. Brief Description of Empirical TEA Study .....................................67
2. Application of ITEAM to VBS2 Based ITE ....................................67
B. DISCUSSION .................................................................................................70
1. VBS2 TEA Results .............................................................................70
2. ITEAM Results...................................................................................71
C. CONCLUSIONS ............................................................................................73
V. CASE STUDY II: FULL SPECTRUM COMMAND ..................................................75
A. METHOD .......................................................................................................75
B. DISCUSSION .................................................................................................81
C. CONCLUSIONS ............................................................................................87
VI. CASE STUDY III: EST 2000 HEAVY WEAPONS TRAINER .................................89
vi ii