DTIC ADA509039: The Importance of Crew Resource Management Behaviors in Mission Performance: Implications for Training Evaluation

21 Pages·0.88 MB·English

by  Defense Technical Information Center

MILITARYPSYCHOLOGY,2003,15(1),77-96
The Importance of Crew
Resource Management Behaviors
in Mission Performance:
Implications for Training Evaluation
RobertT. Nullmeyer
AircrewTrainingResearchDivision
AirForceResearchLaboratory
Mesa, Arizona
V. Alan Spiker
AnacapaSciences
SantaBarbara, California
Cockpit/crewresource management(CRM) trainingwithinthe military has grown
rapidlydespitethepaucityofempiricaldatalinkingCRMtomissionperformance.
CRMtrainingobjectives(andcoursecontent)areoftentoovaguetoallowmeaningful
trainingevaluationwithinthecontextoftraditionaltransfer-of-trainingparadigms.A
multimeasuremethodologythatexploitsallsourcesofarchivalandobservationaldata
withinatrainingorganizationhasthepotentialtoadvancetrainingevaluation,particu
larlyforcrew-basedskillssuchasCRM.ThisarticlediscussesavarietyofCRMdata
sourcesandpresentsfindingsusing2ofthesesources:instructorcommentsinstudent
trainingfoldersandover-the-shoulderobservationsofcrewsintacticalsimulators.In
structorcommentsrevealedthatCRMproblemsearlyintrainingmostfrequentlyin
volvedecisionmakingandcommunicationamongcrewmembers.Over-the-shoulder
observationsofexperiencedcrewsshowedhighcorrelationsbetweenindependentrat
ingsofCRMproficiencyandmissionperformance.Themosteffectivecrewsexhibited
suchcharacteristicCRMbehaviorsasthepresenceofasingleleaderandwillingnessto
changeplansbasedonchangingmissionsituations.Thearticleclosesbydescribing
how these studydatacanbe usedto restructureCRMtrainingintoa setofbehav
ior-basedobjectivesthatwillenablemeaningfulevaluationofitseffectivenessinim
provingtheperformancelevelsofallstudentcrews.
RequestsforreprintsshouldbesenttoRobertT.Nullmeyer,AirForceResearchLaboratory,6030
SouthKentStreet,Mesa,AZ 85212-6061.E-mail:[email protected]
Report Documentation Page Form Approved
OMB No. 0704-0188
Public reporting burden for the collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and
maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information,
including suggestions for reducing this burden, to Washington Headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington
VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to a penalty for failing to comply with a collection of information if it

does not display a currently valid OMB control number.
1. REPORT DATE 2. REPORT TYPE 3. DATES COVERED
01 DEC 2003 Journal Article 01-01-2001 to 30-11-2003
4. TITLE AND SUBTITLE 5a. CONTRACT NUMBER
The Importance of Crew Resource Management Behaviors in
5b. GRANT NUMBER
Mission Performance: Implications for Training Evaluation
5c. PROGRAM ELEMENT NUMBER
62205F
6. AUTHOR(S) 5d. PROJECT NUMBER
Robert Nullmeyer; V. Spiker 1123
5e. TASK NUMBER
5f. WORK UNIT NUMBER
7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PERFORMING ORGANIZATION REPORT NUMBER
Air Force Research Laboratory,Aircrew Training Research AFRL/HEA
Division,6030 South Kent Street,Mesa,AZ,85212-6061
9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR’S ACRONYM(S)
Air Force Research Laboratory, Warfighter Training Research AFRL/RH; AFRL/RHA
Division, 6030 South Kent Street, Mesa, AZ, 85212-6061
11. SPONSOR/MONITOR’S REPORT NUMBER(S)
AFRL-RH-AZ-JA-2003-0002
12. DISTRIBUTION/AVAILABILITY STATEMENT
Approved for public release; distribution unlimited
13. SUPPLEMENTARY NOTES
Published in Military Psychology, 2003, 15(1), 77-96
14. ABSTRACT
Cockpit/crew resource management (CRM) training within the military has grown rapidly despite the paucity of empirical data
linking CRM to mission performance. CRM training objectives (and course content) are often too vague to allow meaningful
training evaluation within the context of traditional transfer-of-training paradigms. A multimeasure methodology that exploits
all sources of archival and observational data within a training organization has the potential to advance training evaluation,
particularly for crew-based skills such as CRM. This article discusses a variety of CRM data sources and presents findings using
two of these sources: instructor comments in student training folders and over-the-shoulder observations of crews in tactical
simulators. Instructor comments revealed that CRM problems early in training most frequently involve decision making and
communication among crew members. Over-the-shoulder observations of experienced crews showed high correlations between
independent ratings of CRM proficiency and mission performance. The most effective crews exhibited such characteristic CRM
behaviors as the presence of a single leader and willingness to change plans based on changing mission situations. The article
closes by describing how these study data can be used to restructure CRM training into a set of behavior-based objectives that
will enable meaningful evaluation of its effectiveness in improving the performance levels of all student crews.
15. SUBJECT TERMS
Crew resource management; Mission performance; Training evaluation; Cockpit resource management; Transfer of training;
CRM; Decision making; Crewmember communication; Crew-based skills; Flight simulators
16. SECURITY CLASSIFICATION OF: 17. LIMITATION 18. NUMBER 19a. NAME OF RESPONSIBLE
OF ABSTRACT OF PAGES PERSON
a. REPORT b. ABSTRACT c. THIS PAGE Public 20
unclassified unclassified unclassified Release
78 NULLMEYERANDSPIKER
Human erroris frequently linked with aviation accidents and incidents (Kayten,
1993). Infact, HelmreichandFoushee (1993) reportedthataircrewactions were
causalfactors inmorethan70%ofhull-lossaccidentsintheworldwidecommer
cialjetfleetfrom 1959through 1989.InRuffellSmith's(1979)landmarksimula
torstudyoncockpitworkload,thefactorsthatmostdifferentiatedeffectivecrews
from weaker ones were leadership, decision making,·and resource management
ratherthanmoretechnicallyorientedskills. Inresponsetofindings suchasthese,
the National Aeronautics and Space Administration (NASA) sponsored a work
shop, Resource Management on the Flightdeck, in 1979 (Cooper, White, &
Lauber, 1980),whichiscommonlyviewedastheoriginofformalcockpit/crewre
sourcemanagement(CRM)training.NASAandtheMilitaryAirliftCommandco
sponsoredafollow-up conferenceonCRMtrainingseveralyearslater(Orlady&
Foushee, 1987). This workshop marked the expansion ofCRM training into the
militaryservices.EachofthemilitaryservicesquicklyaddedCRMtrainingtose
lectedprograms,andwithinafew years,CRMinstructionbecamemandatoryfor
allmilitaryaviators.
This rapid growth ofCRM training throughoutaviationoccurreddespite sur
prisinglylittleempiricalevidencelinkingthistrainingtoimprovedmissionperfor
mance.TheCRMvalidationdatathatdoexistaredominatedbytraineejudgments
about its value (e.g., lIgen, 1999; Salas, Fowlkes, Stout, Milanovich, & Prince,
1999).Despiteits shorthistory, five distinctgenerationsofCRMtrainingcanal
readybedocumentedinthecommercialairlines(Helmreich,Merritt,& Wilhelm,
1999),witheachgenerationrepresentingasubstantialshiftintrainingphilosophy
andcontent.
The military's approach to measuring the effectiveness oftraining interven
tions-be they a revised program ofinstruction, improved courseware, or new
training device-has traditionally followed Kirkpatrick's (1996) four-stage
model,depictedinthemiddleofFigure 1(Bell&Waag, 1998;SalasetaI., 1999).
Althougheffectivenessis ultimatelyequatedwiththe"contributionoftraining to
therequiredavailabilityofcombatpower"(Stage4; Bell& Waag, 1998,p. 234),
the vastmajorityofeffortisfocused ontheearlierstages. Theseare the trainees'
perceived value of the training, the degree to which the to-be-trained knowl
edge-skills-attitudes(KSAs)areactuallylearned,andtoamuchlesserextent,the
availabilityoftargetedKSAsforuseonthejob.Fromaresearchstandpoint,Stage
3isthedefiningfeatureofeffectiveness,asdemonstratedbyapositivetransferto
thejob(combat)environment. Correspondingtoeachstageisatraditional meth
odologyforevaluatingtraining,asshownintheleftpartofFigure1.Theseinclude
a survey oftrainees' attitudes toward their training, a pre- and postexperimental
comparisonoftrainees'performanceontheKSAs,afulltransfer-of-trainingstudy
conductedin thejobenvironment, and a cost-effectiveness analysis that demon
stratesthedesiredorganizationalproductivityorperformanceimpact(e.g.,combat
readiness)atanacceptablecost.
BEHAVIORSANDTRAININGEVALUATION 79
Performance
Validity
TrainingRecords
instructorcomments
Illstrllt:torratings
GainingUnitSurveys
I I
Accidentreports
"Naturalexperiments"
Other
Traditional Kirkpatrick's4-stagemodel
Measurement
Measurement
"Opportunities"
Methods
Wasthetraining Wasthetraining
contentappropriate effectiveinaiding
forthejobandwas jobperformance?
itdeliveredeffectively?
FIGURE1 Frameworkformeasuringtrainingeffectiveness.
Thoughinfluential,themodel'slimitationsasacomprehensivetheoryofeffec
tiveness measurementhavebeennotedbyanumberofresearchers. Forexample,
Bell and Waag (1998) acknowledged the "brute force" aspectofthe model's se
quential stages and cited the need to distinguish between training processes and
performance,particularlyinStage2.Salasetal.(1999)usedthemodel'shierarchy
primarilyas a theoretical "driver" for collectingdatawithin amultimeasurement
frameworkontrainees'reactions,attitudes,knowledge, andbehavior,withnose
quentialprioritiesimposedonthemeasures.
Bell andWaag (1998) and Thurman and Dunlap (1999) reported apaucity of
transferstudiescorrespondingtoStage3ofKirkpatrick's(1996)model.Thisdearth
is tiedtocost, methodologicalandoperationalconstraintsassociatedwithtesting
newlytrained(andnowvaluable)personnelinactualornear-realisticmissionsce
narios,solicitingandpreservingasuitablecontrolgroup,andobtainingtheneces-
80 NULLMEYERANDSPIKER
sarypersonnelstabilitywithinaturbulentenvironment(Thurman&Dunlap,1999).
Barrierstoeffectivetransferaremany, withperhapsthemostsignificantbeingno
cleardefinitionofwhatisbeingtrained.TheKirkpatrickmodelwaspromulgatedin
an era when procedurally oriented KSAs from prescribed task to training lists
(TTLs)formedthecontentfortraining(Bills&Wood,1999).Withtheproliferation
ofsimulationdevicesgearedtowardfacilitatingcognitive-basedprocessessuchas
decisionmakingandsituationawareness,thetask-specificbasisoftrainingcontent
hasbeenaltered.ThishasbeenparticularlytrueinCRM,where---dependingonthe
researcher or the tactical domain-it has been referred to as a training program
(lIgen,1999),instructionalstrategy(SalasetaI., 1999),tacitknowledge,metaskill,
orenterprise(Bills& Wood, 1999).
Indiscussing trainingprogramevaluation, Goldstein(1987)distinguishedbe
tween gauging the "training validity" ofthe intervention, that is, determining if
trainingcontentisappropriateandeffectivelydelivered, and gaugingits "perfor
mancevalidity,"thatis,theextenttowhichtheinterventionfacilitatessubsequent
jobperformance. Asdenotedbythe arrows intheright-handportionofFigure 1,
thesetwoaspectsoftrainingeffectivenesscanbeassessedinparallel,andtheyex
tendacrossallstagesofKirkpatrick's (1996) model.
Giventhelimitationsofthestagemodeldiscussedpreviouslyandtheattendant
difficultiesinconductingtransfer-of-trainingexperiments,webelievethatamore
viableframeworkformeasuringtrainingeffectivenesswillemphasizecapitalizing
onmultiplemeasurementopportunities,muchasisdoneintheprogramevaluation
area(Cook&Campbell, 1979).Consideringthetypicalmilitarytrainingenviron
ment,thereareanumberofalternativedatasourcesthatbeardirectlyontheeffec
tiveness of a unit's training. Five of these opportunities are indicated in the
right-hand portionofFigure 1.Although theseare orderedaccordingtotheirap
proximateplacementwithinKirkpatrick's(1996)stages,eachmethodcanapplyto
eithertrainingvalidityorprogramvalidity;consequently,wehavenotattemptedto
specifyformal links withanyofthestages.
Naturalistic observations ofmilitary aviators during simulator missions have
revealedstrongempiricallinksbetweenthequalityofcrewinteractions(process)
andmissionperformance(outcome).Povenmire,Rockway,Buenecke,andPatton
(1989) observed seven B-52 crews execute a tactically realistic scenario in a
high-fidelity simulatorand reported a statistically significantrank ordercorrela
=
tionbetweenmissionperformanceandCRM(r .83).Practicinginquiryandadvo
cacy,avoidingdistractions,distributing workload,andresolvingconflictsemerged
asfactors thatweresignificantlycorrelatedwithoverallcrewcoordination.
Thornton,Kaempf, Zeller,andMcAnulty(1992)observed 19pairsofaviators
astheyflew acombat-orientedmissioninanadvancedUH-60BlackHawksimu
lator. Aircrewcoordinationwasdefinedintermsoftherate,pattern, content, and
qualityofinteractionsalong 13functionalcategories(inquiry,command,declara
tive, etc.). Mission effectiveness was defined in terms of navigation accuracy,
BEHAVIORSANDTRAININGEVALUATION 81
threatavoidance,andperformanceofanonprecisionapproach. Patternsandtypes
ofcommunicationwererelatedtooutcome-basedindexesofmissionperformance,
butrateofcommunicationwasnot.
Brannick,Prince,Prince,andSalas(1995)alsoreportedastrongempiricalrela
tion between team coordination and performance in a Navy study in which 52
two-personcrewsflew nontacticalscenariosinalow-fidelity,tabletopT-44flight
trainer.Sixteamcoordinationdimensionswererated: assertiveness,decisionmak
ing, adaptability, situationalawareness, leadership, andcommunication. Arating
scaleranging from 1(unacceptable) to 5 (excellent) was developedfor each di
mension,andbehaviorsrepresentingtheperformanceexpectedwereprovidedfor
the scale points. Forexample, assertive behaviors associated with higherratings
includedquestioningsomedirectionsfromairtrafficcontrolandadmittingconfu
sionaboutanassignedaltitude. Allsixprocessdimensionswerepositivelycorre
latedwithperformance, withcorrelationsrangingfrom .43 to.69.
TheremainingthreemethodsinFigure 1illustratehoweffectivelytraininghas
preparedstudentsforperformingtheirdesignatedrolesin theiroperationalunits.
Thesemeasuresincludereviewingthesurveyscompletedbytrainingunitsupervi
sorsconcerninghowwelltraineescouldsupporttheunitwithoutadditionaltraining,
trackingandtrendingaccidentreportstoidentifylinkswithcommonsetsofprior
trainingexperiences,andexaminingtheconductof"naturalexperiments"thatarise
due to operationalcircumstances. An exampleofnatural experimentsinvolveda
comparisonofmissionreadinessduringtheGulfWarforgroupsofaviatorswhoei
therdidordidnotreceivefull-missionsimulatortraining(thesimulatorwasunavail
ablefortechnicalreasonsforacohortofaviators)priortotheirarrivalintheMiddle
East(Rakip, Kelly, Appler, &Riley, 1993).Thatsimulator-trainedMH-53JPave
Lowoperatorsweremoreabletoconductmissionsimmediatelyonarrivalinacoun
trycomparedwiththeirnonsimulator-trainedcounterpartswasconsideredtobeevi
dencesupportingtheeffectivenessoftheMH-53Jflightsimulator.
Ultimately, CRMtrainingeffectivenessevaluationmustaddress the degree to
which actual training needs are satisfied. Competing definitions and poorly de
finedtrainingobjectiveshaveundoubtedlycontributedtothecurrentdearthofem
pirical CRM training studies. The studies reported here were designed to define
CRMinconcrete,observabletermsforMC-l30Pcrewtrainingatthe58thSpecial
OperationsWing,KirtlandAirForceBase,NewMexico.Ourobjectiveswereboth
toestablishthecontentofCRMtrainingandtodevelopmeasuresfor subsequent
trainingevaluation.
The top two boxes on the rightportion ofFigure 1highlight the methods de
scribedin this article. Althougheachisdescribed inmoredetail inthefollowing
sections, analysisoftraining records andnaturalistic observations were methods
ofchoicebecausetheyshedlightontherelevanceofexistingCRMcoursecontent
forMC-l30PcrewsandtherelationofputativeCRMprocessestomissionperfor
mance. By contentanalyzing the instructors' comments in student grade folders
82 NULLMEYERANDSPIKER
forsimulatorandflight-linemissions,wemayascertaintheCRMproblemsexpe
riencedbystudentsintraining.Byaugmentingtheseanalyseswithin-depth,natu
ralistic observations ofcrews during simulator and flight-line training, we may
constructabetterpicture ofwhichCRM behaviors contributeto successfulmis
sionperformanceand, ofthese, whicharepresentlytaughtduringCRMtraining.
STUDY 1: USING STUDENTGRADE FOLDERS
TO ASSESS CRM PROFICIENCY
Overview
Analysis ofarchival training records, such as student grade folders, provides at
least three advantages as a source of training effectiveness data. First, training
records are routinely maintainedby training wings and do notrequire additional
effort to collect. Second, they are populated by input from instructors who are
experts in that student's crew position and who have acquired considerable fa
miliarity witheachstudent'scapabilities. Third, they areoftenquantitativeinna
ture and hence amenable to statistical analysis.
MC-130P StudentGrade Folders
Grade folders are maintained for eachMC-l30P student as he or she completes
missionqualification(MQ)training.Foreachacademic,simulator,andflight-line
= =
trainingsession,theinstructorassignsalettergrade(P proficiencyadvance, E
exceptional, S = satisfactory, T= needs training, U = unsatisfactory, I = incom
plete)inthestudent'sForm15,AircrewTrainingRecord.Tobeausefulsourceof
proficiencydata,ameasuremustvaryacrossstudents;otherwise,onecannotinfer
theimpactofprogramvariablesortraininginterventions.However,inouranalysis
ofMC-l30Pstudentrecords,morethan98%ofstudentgradesintheForm15were
assignedan"S."
MQtraining progresses through seven simulatortrainingevents: aconversion
mission,twoday-tacticalmissions,andfournight-tacticalmissions.Eachblockof
simulatortrainingisfollowedbyflight-lineinstructionofcorrespondingcomplex
ity. To keep track ofthis progression, MC-130P instructors fill out a Form 14,
Aircrew Training Progress Record, aftereach simulator and flight-line mission.
This preprintedform provides asetofrequiredproficiency levels (RPLs) for the
trainingeventsassociatedwiththatmissionprofile.Eventsaretaskbased,suchas
airdropchecklist, simulatedenginefailure, nightvisiondeviceoperations,andso
forth. Performanceand knowledgearegradedon a4-pointscale, ranging from 1
(extremely limited) to 4 (highly proficient). As the student progresses through
training, theRPLs fortheeventsineachtrainingprofilebecomemorestringent.
BEHAVIORSANDTRAININGEVALUATION 83
Instructors cross offRPLs onthepreprintedform aseachtrainingeventis ac
complished.IfastudentexceedsorfailstomeetanRPL,theinstructormustwrite
in the actuallevelofperformance orknowledge demonstrated; however, instruc
torsrarelynotedeviationsfromtheRPLbecausetheyarebusyanddonotwishto
havestudentdeficienciesnotedinthepermanentrecord.Consequently,analysisof
theRPLdata,suchasaggregatingthenumberofsub-RPLevents,willnotproduce
usefulmeasuresofproficiencybecausemostentriesaretheunannotatedRPLs.
Inadditiontogrades,instructorswritecommentsfollowingeachsimulatorand
flight session on Form 13, Training Comments Record. The comments are un
structuredandarenotnecessarilytiedtotherequireditemscoveredintheForm14.
These comments provide information that is potentially a rich source ofprofi
ciencydataasinstructorsarefreetoexpresstheirreservationsregardingastudent,
knowing that theirremarks are not reflected in the student's recorded grade. In
structors may also laud exemplary performance and can go over the comments
withthestudentafteratrainingmission, usingitas ateachingordebriefingaid.
In analyses of MC-130P crew training records for each position, we deter
mined that instructors make extensive comments that, when aggregated across
missions, can be reliably classified into positive and negative cases. Moreover,
the comments can be sorted into functional categories characteristic of each
crew position (e.g., crew coordination, equipmentknowledge). This analysis re
vealed that instructor comments are quite specific (e.g., "missed several radio
calls," "must keep checklist flowing to ensure proper crew responses," "need
more positive continuous guidance to pilot") and yield valuable insights con
cerning areas where student proficiency is strong or weak. To the extent that
thesecommentsarerecordedroutinelyandcomprehensively,theycanbecontent
analyzed, aggregated, and quantified to yield data-based assessments ofstudent
proficiency and, ultimately, training effectiveness.
Method
Basedonthepreliminaryanalyses notedpreviously, we usedthe instructorcom
mentsfrom studentgradefolders toassess theeffectivenessofthe wing'scurrent
CRM training. We reviewed arepresentative sample of20records from the five
crewspecialtiesintheMC-130P:pilots,navigators,flightengineer(FE),commu
nicationsystemsoperator(CSO),andloadmaster(LM).Wethenenlistedthehelp
oftwosubjectmatterexperts(SMEs)toreviewtherecordsaccordingtothemeth
odology outlinedin thefollowing paragraph. Both wereexperiencedinstructors,
oneinairbornecommandandcontrolaircraftandtheotherinspecialoperations,
fixed-wing aircraft.
Working independently, the SMEs reviewed each training record and high
lightedallinstructorcommentsrelevanttoCRM.Thesecommentswerethenpara
phrasedandtranscribedontoafour-page, structuredTrainingRecordEvaluation
84 NULLMEYERANDSPIKER
Worksheet.TheworksheetwasorganizedaroundthesixCRMareascurrentlycov
eredinAirForceInstruction11-290(Cockpit/CrewResource, 1998).TheseCRM
areas are mission planning and debrief, task management, situation awareness,
crew coordination, communication, and risk management decision making. We
addedaseventh area, tacticsemployment,toaddressthecombat-intensiveopera
tions requiredofthis aircraft. Eachparaphrasedcommentwasplacedin therele
vantCRMcategoryandthenratedona5-pointscale,rangingfrom 1(significantly
belowexpectations)to5(exceptional),withamidpointof3(levelexpectedforthis
leveloftraining). Oncecomments were transcribed andrated, a summaryrating
forthatCRMcategory was assigned. Finally, an overallproficiencyrating, again
ona 1to5scale,wasassignedforeachstudent.
Results
Interraterreliability. Thefirstgoaloftheanalysiswastodetermineifthetwo
SMEswereconsistentintheirratingassignments.Thecorrelationbetweenthetwo
setsof20overallproficiencyratings was .81, within therecommendedrange for
acceptable interrater reliability (Cronbach, 1990). Looking at the ratings them
selves, wefound that the two raters producedidenticalratings for 16ofthe stu
dentsanddifferedbyonlyonepointfortheotherfourstudents.Wethusconclude
thatSMEscanreliablyassignproficiencyratings basedonthecommentsthatin
structorsplaceinstudentgradefolders.
Inferringstudentproficiency. Havingestablishedthereliabilityoftherating
process,wehadoneoftheSMEsextendhisreviewtoincludeallMC-130Pstudent
recordsfrom1998,atotalof87records.Wethenexaminedtheabsolutevaluesofthe
ratingstodetermineiftherewassufficientvariationacrossstudentstoassessCRM
training.Ofthe87records,morethanonethirdreceivedaratingotherthan3,amuch
higherpercentagethanwasevidentintheinstructorgradesontheForm 14s.
Even greater sensitivity is seen when the comment-based proficiency ratings
arebrokendownbyCRMcategory,asshowninFigure2.Computingtheaverage
ratingvariabilitywithineachcategory,wedeterminedthatascaledifferenceof.14
canbeconsideredstatisticallymeaningful(Hays, 1973).Onthatbasis,weseethat
mission preparation and crew coordination received substantially above-average
ratings (3.2-3.3), with decision making and communications (2.8-2.9) signifi
cantlybelowaverage.
Qualitativeanalysis. Having identified the CRM categories that stand out
statistically,wecanexaminetheassociatedinstructorcommentstopinpointareas
wherepresentCRMbehaviorsarestrongandweak. Inperformingthisanalysis,it
should be noted that such comments have two aspects: evaluative and directive.
Theevaluativecomponentisusedtogaugestudentproficiencyinthecommented
areaandistypicallyrepresentedbyanadjective,suchas"good"missionplanning,
BEHAVIORSANDTRAININGEVALUATION 85
5 ~
4 -
3 - - avg=3.1
Avg ~
Profi- 2 l IF
ciency 1 I--
Jli
Rating
1'\. ii
~
MP CC SA TM Com DM TE
CRM
FIGURE2 AveragestudentproficiencybyCRMcategory.CRM=cockpit/crewresource
= = =
management; MP mission planningand debrief; CC crewcoordination; SA situation
= = =
awareness;TM taskmanagement;COM communication;DM riskmanagementdecision
making;TE=tacticsemployment.
"excellent"missionbriefing,or"weak"situationawareness.Thedirectiveaspects
ofeachcommentletus extractthe specific crew behaviors that wereeitherdefi
cientorlaudable.Bydirective,wemeansuchcommentsas"slowtopreparebrief,"
"needstothinkfurtheraheadoftheaircraft,"and"missedradiocalls."Thesecom
ments areusuallygiven tothe studentas verbalfeedback duringthetrainingses
siontopromoteimmediateimprovementortoreinforcesomeessentialskill.Over
the long term, the content of these comments can be collected, analyzed, and
foldedbackintoanimprovedtrainingcurriculumas asetoftargetbehaviors.
Toillustrate, wereviewedthecompiledsetofinstructorcommentsin mission
preparationand crew coordinationto ascertainwhy these categories stoodoutas
positiveinstancesofCRM.Forthemostpart,wesawthatinstructorswerecompli
mentingstudentsonsuchaspectsas"thorough"planning,"concise"briefings,and
"goodbackingupcrewmembers."Theseareasareemphasizedinthepresenttrain
ingandseemtohavebeeninternalizedbythestudents. Yettherewerealsonega
tivecommentsthatindicatedareasinneedofimprovement,suchasnavigationand
leadership.Exampleshereincludetheneedforprovidinggreaterannotationofsig
nificantterrainfeatures onmaps,discussingmoreobstaclesinthelow-levelbrief,
andtakingfirmercontrolofthecrew.
Turning to the weaker CRM areas, decision making exhibited a wide assort
mentofdeficiencies thatprimarily involved pilots and navigators. A majordefi
ciencyentailedslowreactions toconditionsrequiring morerapidjudgment,such
as initiatingemergencyprocedures,respondingto lossofengine,turning tofinal
approach, joining up during aerial refueling, correcting the flight profile, and
breakingoffformation duringthe onsetofinstrumentmeteorologicalconditions.
Ahostofcommunicationproblemareaswerealsoexposedforallcrewmembers.