New Agendas for Human-computer Interaction: A Special Double Issue of human-computer Interaction (Human-Computer Interaction, Vol 15, Nos. 2 & 3)

194 Pages·2001·1.22 MB·English

by  Wendy A. Kellogg

HUMAN-COMPUTER INTERACTION, 2000, Volume 15, pp.69–74
Copyright © 2000, Lawrence Erlbaum Associates, Inc.
Introduction to This Special Issue
on New Agendas for
Human–Computer Interaction
Wendy A. Kellogg, Clayton Lewis,
and Peter Polson
IBM T.J. Watson Research Center
University of Colorado
ThefivearticlescontainedinthisspecialissueofHuman–ComputerInterac-
tion(HCI)wereoriginallypresentedataworkshopatSnowMountainRanch,
Colorado,inFebruary1999.Thatworkshopmarkedboththe10thanniver-
saryoftheHuman–ComputerInteractionConsortium(HCIC)andtheendof
themillennium.Accordingly,theworkshoptheme,“Human–ComputerIn-
teractioninthe21stCentury:ProspectsandVisions,”calledonparticipantsto
takestockandlookforwardtonewdirectionsforHCI.Theresults,wethink,
are thought provoking.
Togetaperspectiveontheevolutionofthefield,itisusefultorevisitthean-
alog of this special issue—an earlier HCI special issue, Foundations of Hu-
man–ComputerInteraction(Card&Polson,1990)—whichoriginatedfromthe
firstmeetingoftheHCICin1989.Indoingsowearestruckbytwoobserva-
tions.First,someoftheworkintheearlierissuehasprovedtobefoundational;
indeedsomeofthearticlesarecitedinthisissue.Second,thecomparisonre-
vealshowthefoundationsofthefieldhaveshifted,inthatissuesofmajorfocus
in this issue were barely hinted at in 1990. In particular, the two collections
clearlyshowthegrowingawarenessthatdesignfortheisolatedindividualuser
isinadequatetoproducesystemsofpracticalvalueinsupportingtherealityof
people working together.
All of the articles in this special issue reflect the field’s evolution toward
morecomplexandmorecontextualizedviewsofinteractivesystemsandtheir
70 KELLOGG, LEWIS, POLSON
use.Thisconsiderationofmorerealistictasks,models,andsituationsofuse
canbeseeninthevariousdomainsofinquiryrepresentedhere,fromtask-ac-
tionmodelsintheGOMSfamilytosupportforcollaborativework.Itisalsoan
impetus for more sweeping proposals to rethink how HCI research is orga-
nizedandhowitrelatestootherareasofscience.Thearticleshereincollec-
tivelyofferasetofproposalsforthefuture:newagendasthatcantakeresearch
in HCI and Computer-Supported Cooperative Work (CSCW) to the next
level.
Thesearticlesareatthesametimeconservativeandradical.Theyarecon-
servativeinthattheyareskepticalofthepopularhypeprevailingatthecen-
tury’s end, the grandiose visions of an all-powerful technological
transformationofexistence.Theyareradicalinthatallsharplyquestionpre-
vailing methods and approaches in HCI and CSCW.
TheconservativethemecanbeheardclearlyinAckerman(thisissue).Con-
trarytomillennialoptimism,hearguesthatweremainunabletodevelopsuc-
cessfultechnologicalsupportforsocialprocesses.Similarly,OlsonandOlson
(this issue) argue that the media-touted view of communication technology
erasing distinctions of time and space is seriously in error. At another level,
BhavnaniandJohn(thisissue)warnthatharnessingnewtechnologyrequires
farmorethancreatingausableorlearnableinterfaceandputtingitinfrontof
people who will then absorb and master it with little effort.
The radical theme is also discernable across these articles. Whittaker,
Terveen, and Nardi (this issue) are strongly critical of prevailing research
practice in HCI, building on the earlier critique of Newman (1994) that
called attention to the disproportionate emphasis on radical innovation
ratherthanevolutionaryimprovementinthefield.BhavnaniandJohn’s(this
issue)attackontheincompletenessoflearnabilityandusabilityasgoalsfor
system design strikes not only at attitudes toward technology, but also at
common thinking within the HCI research community. Both Olson and
Olson(thisissue)andAckerman(thisissue)callforsignificantreorientations
ofresearch;techno-hypemustbesupplantedbyhardworkonadeepintel-
lectual agenda. Finally, Furnas (this issue) boldly outlines a theoretical
frameworkthatchallengesHCIresearcherstothinkseriouslyandsystemati-
callyaboutthemultifariouscontextualinteractionsthatwilldefinethesuc-
cessorfailureofsystems.
Althoughthesethemestiethearticlestogether,eacharticlealsopresentsa
specificcritiqueofthefieldandanagendaformovingtheworkforwardinnew
ways. Here is a preview of the arguments.
Whittakeretal.(thisissue)urgethattheHCIresearchcommunityborrow
thereferencetaskapproachthathasbeeneffectiveinorganizingresearchin
speechrecognition.TheyarguethatHCIresearchsofarhassufferedfromdis-
persion,witheachworkerorresearchgroupsettinganagendaonlyweaklyre-
INTRODUCTION 71
lated to what others are doing. Under the Whittaker et al. proposal, HCI
workerswouldproposeandagreeonarelativelysmallsetofcommonprob-
lems,embodiedinsampleusertasks,andwouldfocuseffortonprovidingim-
provedsupportforthese.Overtime,thestandardofaccomplishmentinthe
fieldwouldriseasgroupssucceedinbetteringthemeasuredqualityofearlier
solutions.
Thereisnoquestionthat,ifthisproposalisadoptedbyaquorumofwork-
ers,theHCIlandscapeofthefuturewillbeverydifferentfromtoday’s.How-
ever, questions abound. Do we understand any important interactive tasks
well enough to propose performance measures adequate to guide progress
overanextendedperiod?Willtasksremaincomparableovertime,giventhe
rapid evolution of technologies?
BhavnaniandJohn(thisissue)showthatusersofcomplexapplicationsdo
notspontaneouslyusestrategiesthatmakeuseofoperationsuniquetocom-
puters(e.g.,definingaggregationsofobjectsandthenusingasinglecommand
tooperateonallobjectsintheaggregate).TheyuseGOMSmodelstoshow
that these strategies define methods that have very different structures from
typical GOMS methods (e.g., a method to delete a word).
Theseresultshavefundamentalimplications.Landauer(1995)arguedthat
computer applications have not improved the productivity of knowledge
work because flawed product development processes produce applications
with poor user interfaces. Although his conclusion is certainly justified,
BhavnaniandJohn(thisissue)showthateveniftheinterfacetoacomplexap-
plicationisrelativelygood,usersdonotspontaneouslylearnandusestrategies
that would make them far more productive. In addition, strategies useful in
complexapplicationsarenotdescribedintrainingandreferencedocumenta-
tion.Thedominantparadigmforusertrainingistoprovidealimitedintroduc-
tion to the user interface and basic functionality of a complex application.
Usersareexpectedtoacquireadvancedskillsastheygainmoreexperience
withanapplication.Minimalistdocumentation(Carroll,1990)anduserinter-
facesthatsupportlearningbyexplorationarewell-known,successfulmethod-
ologies designed to support this training paradigm. Bhavnani and John’s
resultsshowthatthisparadigmhasfundamentallimitations;itwillnotsupport
thedevelopmentofthekindsofpowerfulstrategiestheydescribe.However,
userscanbetrainedtousethesestrategies,andtheypresentpreliminaryevi-
dencethatthesestrategiesareretainedandgeneralized.Thus,theyarguethat
theagendaforHCIneedstoshiftawayfromsupportforlearningbyexplora-
tion to support for training users on productive strategies.
Widespread deployment of organizational intranets and access to the
Internet has led to expectations that individuals and teams will be able to
seamlesslycollaborateremotely.In“DistanceMatters,”OlsonandOlson(this
issue)arguethatthissimplisticviewiswrong.Theyreviewover10yearsof
72 KELLOGG, LEWIS, POLSON
fieldandlaboratoryinvestigationsofcollocatedandnoncollocatedsynchro-
nousgroupcollaborations,providingasignificantnewanalysisoffactorscon-
tributing to a group’s ability to collaborate over distances and an outline of
whatmaybepossibleinthefutureastechnologyevolvesandusers’andorga-
nizations’sophisticationintheuseoftechnologyandcollaborationincreases.
Inparticular,theyidentifyfourkeyconcepts:commonground,thecoupling
of work, collaboration readiness, and collaboration technology readiness.
Thisanalysisframesthediscussionofdistantcollaborationintermsthatrange
fromtheartifactsandgroupwarethatsupportcollaborationtocomplexand
subtlefactorsincludingcontextandtrust,timezones,andcultures.Although
theOlsonsremainskepticalaboutthepotentialforeffortlessremotecollabo-
ration—nomatterhowgoodthetechnologybecomes—theirfocusedreviewof
alargebodyofempiricalevidenceandanalysisofthecriticalfactorsinfluenc-
ing distance collaboration sets the stage for future work in this area.
Ackerman(thisissue)takesskepticismtothenextlevelbypointingoutthat
althoughtechnicalsystemsarestilltoobrittleandrigidtoaccommodatethe
subtletyofsocialprocesses,thisisnolongerduetoignoranceonthepartof
technologistsabouttherelevantsocialprocessestosupport.Hecallsintoques-
tionwhetheranyamountofunderstandingofsocialrequirementscanleadto
thecreationofsufficientlyresponsivesystems.Dubbingthisthe“social–tech-
nicalgap,”Ackermanarguesnotonlythatitmaybeimpossibletoabolishbut
also that embracing and addressing it is the heart and soul of CSCW’s mis-
sion—its primary intellectual challenge. After reviewing seminal findings in
CSCW,heconsideredthegapinmoredetailthroughanexamplethatfocuses
onprivacyininformationsystems.Thisleadstoadiscussionofpotentialreso-
lutions for the social–technical gap and a proposal to adapt Simon’s (1996)
ideaofascienceoftheartificialasaframeworkforfutureworkinCSCW.The
proposalisprovocative,suggestingthatthewayforwardistorecognizetheso-
cial–technicalgapasfundamentalandtheprospectforbuildingasciencethat
bridges the gap as the central intellectual mission of the field.
Goingevenfurther,Furnas(thisissue)outlinesanintellectualframework
thattiesHCIresearchintobroadthemesthatrunthroughmuchorallofsci-
ence.HedevelopsananalysisofinteractingsystemsthatpermitsissuesinHCI
tobeidentifiedwithissuesgeneraltoadaptivesystemsofallkindsinbiology,
economics,ormanagement.ThekeyabstractionistheMoRAS,amosaicof
responsiveadaptivesystems.Inthemosaic,thesuccessorfailureofonesub-
systemistiedtothepatternofinteractionswithothersubsystems.Information
technologyhastobeunderstoodnotonlyassupportingsomekindsofsubsys-
temsbutalsoasalteringthewaysubsystemsarecoupled.Understandingthe
coupling is crucial to successful design.
SituatingHCIintheMoRASmeansnotonlythat,forexample,economic
perspectivesarerelevanttoHCIbutalsothatsomeproblemsstudiedinHCI
INTRODUCTION 73
canbeseenasfundamentallythesameassomeproblemsstudiedineconom-
ics.Thisisaboldprogram,onethatprovidesafruitfulbasisforawholerange
of new collaborations for HCI.
Takentogether,thesearticlesdoaremarkablejoboftakingstockofwhere
weareandlookingforwardtowhatcomesnext.Weapplaudtheauthorsfor
theirwillingnesstodothehardworkoflookingbackandforwardinthesame
momentandfortheircourageinrecommendingthesenewdirectionstoallof
usforthebenefitofthefield.Wehopeyouwillfindtheirthoughtsasstimulat-
ingaswehaveandfindyourselfreflectingonhowtheirproposalscouldaffect
your own area of work.
NOTES
Acknowledgments. We thank the reviewers who participated in the process of
bringingthisspecialissuetofruition,aswellasTomMoran,EditorofHuman–Com-
puter Interaction. This creation of this special issue was closely coordinated with a
similar“millennium”specialissueoftheACMTransactionsonComputer–Human
Interaction,editedbyJohnM.Carroll.
REFERENCES
Card,S.K.,&Polson,P.(Eds.).(1990).Foundationsofhuman–computerinteraction
[Specialissue].Human–ComputerInteraction,5(2–3).
Carroll,J.M.(1990).TheNurnbergfunnel:Designingminimalistinstructionforpractical
computerskill.Cambridge,MA:MITPress.
Landauer,T.(1995).Thetroublewithcomputers:Usefulness,usability,andproductivity.
Cambridge,MA:MITPress.
Newman,W.(1994).ApreliminaryanalysisoftheproductsofHCIresearch,using
proformaabstracts.ProceedingsoftheCHI’94ConferenceonHumanFactorsinCom-
putingSystems,278–284.NewYork:ACM.
Simon,H.A.(1996).Sciencesoftheartificial(3rded.).Cambridge,MA:MITPress.
ARTICLES IN THIS SPECIAL ISSUE
Ackerman,M.S.(2000).TheintellectualchallengeofCSCW:Thegapbetweenso-
cial requirements and technical feasibility. Human–Computer Interaction, 15,
179–203.
Bhavnani,S.K.,&John,B.E.(2000).Thestrategicuseofcomplexcomputersys-
tems.Human–ComputerInteraction,15,107–137.
Furnas,G.W.(2000).FuturedesignmindfuloftheMoRAS.Human–ComputerInter-
action,15,205–261.
74 KELLOGG, LEWIS, POLSON
Olson,G.M.,&Olson,J.S.(2000).Distancematters.Human–ComputerInteraction,
15,139–178.
Whittaker,S.,Terveen,L.,&Nardi,B.A.(2000).Let’sstoppushingtheenvelope
andstartaddressingit:AreferencetaskagendaforHCI.Human–ComputerInterac-
tion,15,75–106.
HUMAN-COMPUTER INTERACTION, 2000, Volume 15, pp.75–106
Copyright © 2000, Lawrence Erlbaum Associates, Inc.
Let’s Stop Pushing the Envelope
and Start Addressing It:
A Reference Task Agenda
for HCI
Steve Whittaker, Loren Terveen,
and Bonnie A. Nardi
ATT Labs–Research
ABSTRACT
Weidentifyaproblemwiththeprocessofresearchinthehuman–computerinterac-
tion (HCI) community—an overemphasis on “radical invention” at the price of
achievingacommonresearchfocus.Withoutsuchafocus,itisdifficulttobuildon
previouswork,tocomparedifferentinteractiontechniquesobjectively,andtomake
progressindevelopingtheory.Theseproblemsattheresearchlevelhaveimplica-
tionsforpractice,too;asresearchersweoftenareunabletogiveprincipleddesign
advice to builders of new systems. We propose that the HCI community try to
achieveacommonfocusaroundthenotionofreferencetasks.Weofferargumentsfor
theadvantagesofthisapproachaswellasconsiderpotentialdifficulties.Weexplain
SteveWhittakerisacognitivepsychologistwithinterestsinthetheory,design,
and evaluation of collaborative systems and speech access; he is a senior re-
search scientist in the Human Computer Interaction Department of AT&T
Labs–Research,FlorhamPark,NJ,USA.LorenTerveenisacomputerscien-
tistwithinterestsinrecommendersystemsandonlinecommunities;heisare-
search scientist in the Human Computer Interaction Department of AT&T
Labs–Research, Florham Park, NJ, USA. Bonnie Nardi is an anthropologist
withaninterestinsocialnetworksandactivitytheory;sheisaresearcherinthe
HumanComputerInteractionDepartmentofAT&TLabs–ResearchinMenlo
Park,CA,USA.
76 WHITTAKER, TERVEEN, NARDI
CONTENTS
1. THE PROBLEMS WITH HUMAN–COMPUTER INTERACTION AS
RADICAL INVENTION
1.1. RadicalInventionIsNotAlwaysEffective
1.2. WhatWeDon’tKnow:Requirements,Metrics,andUsesofEveryday
Technologies
1.3. HowWeDon’tKnowIt:TheDisseminationProblem
2. THE REFERENCE TASK SOLUTION
2.1. ReferenceTasksinOtherDisciplines
SpeechRecognition(TheDARPAWorkshops)
InformationRetrieval(TheTRECConferences)
DigitalLibraryandMachineLearning
2.2. LessonsFromDARPAandTREC
CriteriaforSelectingReferenceTasks
PotentialObjectionstoOurProposal
3. HOW TO DEFINE A REFERENCE TASK
4. AN EXAMPLE REFERENCE TASK: BROWSING AND RETRIEVAL
IN SPEECH ARCHIVES
4.1. SelectingandSpecifyingReferenceTasksintheDomainofSpeech
Archives
4.2. DefiningMetrics
4.3. Task-OrientedEvaluationofaSpeechBrowsingSystem
4.4. GeneralIssuesArisingFromReferenceTask-BasedEvaluation
5. CONCLUSIONS
howreferencetaskshavebeenhighlyeffectiveinfocusingresearchintoinformation
retrievalandspeechrecognition.Wediscusswhatfactorshavetobeconsideredinse-
lectingHCIreferencetasksandpresentanexamplereferencetask(forsearching
speecharchives).Thisexampleillustratesthenatureofreferencetasksandpointsto
theissuesandproblemsinvolvedinconstructingandusingthem.Weconcludewith
recommendationsaboutwhatstepsneedtobetakentoexecutethereferencetaskre-
searchagenda.Thisinvolvesrecommendationsaboutboththetechnicalresearch
thatneedstobedoneandchangesinthewaythattheHCIresearchcommunityop-
erates.Thetechnicalresearchinvolvesidentificationofimportantusertasksbysys-
tematicrequirementsgathering,definitionandoperationalizationofreferencetasks
andevaluationmetrics,andexecutionoftask-basedevaluation,alongwithjudicious
useoffieldtrials.Perhapsmoreimportant,wehavealsosuggestedchangesincom-
munitypracticethatHCImustadopttomakethereferencetasksideawork.Wemust
createforumsfordiscussionofcommontasksandmethodsbywhichpeoplecan
comparesystemsandtechniques.Onlybydoingthiscanthenotionofreference
tasksbeintegratedintotheprocessofresearchanddevelopment,enablingthefield
toachievethefocusitdesperatelyneeds.
REFERENCE TASK AGENDA 77
1. THE PROBLEMS WITH HUMAN–COMPUTER
INTERACTION AS RADICAL INVENTION
Researchinhuman–computerinteraction(HCI),particularlyasembodied
intheCHIconference,focuseslargelyonnovelproblemsandsolutionsthat
pushthetechnologyenvelope.Mostpublicationsdescribenoveltechniques
ornovelapplicationsofexistingtechniques.AstudybyNewman(1994)pro-
videdquantitativeevidenceforthis.HecomparedCHIwithfiveotherengi-
neeringresearchfields,suchasthermodynamicsandaerodynamics.Heused
contentanalysistoclassifyabstractsofpublishedarticlesintermsofthetypeof
contributiontheymadetothefield.Hefoundthatinotherengineeringdisci-
plines,over90%ofpublishedresearchbuiltonpriorwork.Therewerethree
majorwaysthatresearcheffortscouldextendpublishedwork:(a)bettermod-
elingtechniques(usedformakingpredictionsaboutdesigns),(b)bettersolu-
tions (to address previously insoluble problems), and (c) better tools and
methods(toapplymodelsorbuildprototypes).Thepicturewascompletely
different for HCI. Newman (1994) conducted a similar analysis of CHI ab-
stractsforthe5yearsfrom1989to1993,attemptingtoclassifyabstractsasde-
scribing one of the three types of enhancements previously identified.
However, only about 30% of articles fit into these categories of developing
prior work. The majority of CHI articles either reported “radical” solutions
(new paradigms, techniques, or applications) or described experience and
heuristics relating to radical solutions.
1.1. Radical Invention Is Not Always Effective
ThisanalysisstronglysuggeststhatCHIisdifferentfromotherengineering
researchdisciplines.But,isthisgoodorbad?Isitaproblemthatourfieldis
dominatedbyattemptsatradicalinvention,apparentlycrowdingouttheprac-
ticeof“normalscience”(Kuhn,1996)?Orisitavirtue?Weofferarguments
thatthecurrentstateofaffairsisproblematicbasedontwodifferentcriteriafor
success in our field.
One criterion for success that is consistent with the radical invention ap-
proachistechnologytransfer.Astrongmotivationforconstantinnovationisthe
exampleofwholenewindustriesbeingcreatedbyuserinterfaces(UIs).Peo-
ple are aware that applications such as Visicalc and Lotus® 1-2-3 drove the
early PC market, and Mosaic/Netscape® led to the Web explosion. In this
view, HCI research is an engine room from which novel interaction tech-
niquesaresnatchedbywaitingtechnologycompanies;orbetteryet,research-
ers start their own companies. There are undoubtedly successes originating
fromwithintheHCIcommunity,includingUItoolkitsandgeneralprogram-
mingtechniques(Rudisill,Lewis,Polson,&McKay,1996),aswellastheideas