Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics (Interactive Technologies)

318 Pages·2008·6.27 MB·English

by  Thomas Tullis

To Susan, Cheryl, and Virginia
– Tom
To my mother, Sara Albert
– Bill
About the Authors
Thomas S. (Tom) Tullis is senior vice president of User Insight at Fidelity
Investments.HejoinedFidelity in1993andwas instrumentalinthedevelopment
of the company’s User Experience department; its facilities include a state-of-the-
art Usability Lab. Prior to working at Fidelity, Tom held positions at Canon
Information Systems, McDonnell Douglas, Unisys Corporation, and Bell Labora-
tories.HeandFidelity’susabilityteamhavebeenfeaturedinanumberofpublica-
tions, including Newsweek, Business 2.0, Money, The Boston Globe, The Wall
Street Journal, and The New York Times. Tullis received a B.A. from Rice Uni-
versity, an M.A. in experimental psychology from New Mexico State University,
andaPh.D.inengineeringpsychologyfromRiceUniversity.Duringhis30yearsof
experienceinhuman–computer interface studies,hehaspublishedmorethan 50
papersinnumeroustechnicaljournalsandhasbeenaninvitedspeakeratnational
andinternationalconferences.TomalsoholdseightU.S.patentsandisaninstruc-
tor in the Human Factors in Information Design Program atBentleyCollege.
William(Bill)AlbertiscurrentlyadirectorofUserExperienceatFidelityInvest-
ments.PriortojoiningFidelity,hewasaSeniorUserInterfaceResearcheratLycos
and Post Doctoral Research Scientist at Cambridge Basic Research. Over the past
decade,Billhasusednearlyeverytypeofusabilitymetricaspartofhisresearch.He
has published more than 20 papers and has presented his research at many
professional and academic conferences. Albert has been awarded prestigious
fellowships through the University of California and the Japanese government for
his research in human factors and spatial cognition. He received B.A. and M.A.
degrees from the UniversityofWashington and aPh.D. from Boston University.
vi
Preface
Userexperience,orUXasit’softenabbreviated,referstoallaspectsofsomeone’s
interaction with aproduct, application,or system. Many people seem to think of
theuserexperienceassomenebulousqualitythatcan’tbemeasuredorquantified.
Wethinkitcanbe.Andthetoolsformeasuringitareusabilitymetricssuchasthe
following:
n How long does it take usersto make a flight reservation using a travel
website?
n How many errors dousers make in trying to log onto a system?
n How many users are successfulwhen trying to instruct their digital video
recorder that they want to record all episodes oftheirfavorite TV show?
n How many users getinto a new ‘‘destination-based’’ elevator withoutfirst
choosingtheir desired floor, onlyto discoverthere are no floor buttons?
n How many users fail to notice the link on a web page that would let them
do exactly what theywant with only one click?
n How many users getfrustrated trying to read the tiny serial number on the
back oftheir new MP3player when registering it?
n How many users are delighted byhow easy itwas toassemble their new
bookcase that came with wordless instructions?
Theseareallexamplesofbehaviorsandattitudesthatcanbemeasured.Somemay
beeasiertomeasurethanothers,buttheycanallbemeasured.Tasksuccessrates,
task times, number of mouse clicks or keystrokes, self-reported ratings of frustra-
tionordelight,andeventhenumberofvisualfixationsonalinkonawebpageare
allexamplesofusabilitymetrics.Andthesemetricscangiveyouinvaluableinsight
into the user experience.
Why would you want to measure the user experience? The answer is, to help
you improve it. With most consumer products and websites these days, if you’re
not improving, you’re falling behind. Usability metrics can help you determine
where you stand relative to your competition and help you pinpoint where you
should focus your improvement efforts—the areas that users find the most con-
fusing, inefficient, or frustrating.
This book is a how-to guide, not a theoretical treatise. We provide practical
advice about what usability metrics to collect in what situations, how to collect
them,howtomakesenseofthedatausingvariousanalysistechniques,andhowto
present the results in the clearest and most compelling way. We’re sharing prac-
tical lessons we’ve learned from our 40+ combined years of experience in this
field.
MeasuringtheUserExperienceisintendedforanyoneinterestedinimproving
the user experience for any type of product, whether it’s a consumer product,
computer system, application program, website, or something else altogether. If
it’s something people use, then you can measure the user experience associated xv
xvi Preface
withit.Thosewhoareinterestedinimprovingtheuserexperienceandwhocould
benefit from this book come from many different perspectives and disciplines,
including usability and user experience (UX) professionals, interaction designers,
informationarchitects,productdesigners,webdesignersanddevelopers,software
developers, graphic designers, marketing and market-research professionals, as
well as projectand product managers.
We hope you find this book helpful in your quest to improve the user experi-
enceforyourproducts.We’dliketohearaboutyoursuccesses(andfailures!).You
can contact us through our website, www.MeasuringUserExperience.com. You
will also find supplementary material there, such as the actual spreadsheets and
graphs for many of the examples in this book, as well as information about tools
that can help inmeasuring the user experience.
* **
The material, examples, and case studies presented in this book have been
included for their instructional value. The publisher and the authors offer
NO WARRANTY OF FITNESS OR MERCHANTABILITY FOR ANY PARTICULAR
PURPOSE and do not accept any liability with respect to the material, examples,
and case studies.
The opinions expressed in this book are those of the authors and do not
representanyofficialpositionorendorsementbytheiremployerorthepublisher.
Acknowledgments
First of all, we thank Diane Cerra, Denise Penrose, Mary James, and Marilyn Rash
fromMorganKaufmann.Yougaveusmanypracticalsuggestionsandpushedusat
alltherighttimes.WealsothankJimLewisandJoyceOghiformanythoroughand
timely reviews of the manuscript. We owe a debt of gratitude to all the authors
whocontributedcasestudies:HoaLoranger,BobBailey,CariWolfson,JaniceNall,
JimLewis,ScottWeiss,ChrisWhitby,AgaBojko,andToddZazelenchuk.Thebook
isfarmoreusefulbecauseofyourwillingnesstoshare yourexperienceswithour
readers.
* **
I would like to thank my wife, Susan, and my daughters, Cheryl and Virginia, for
your support and for putting up with my long evenings and weekends at the
computer! I’d especially like to thank Susan for your reviews and comments on a
numberof the chaptersin this book. You’ve helped me become abetter writer.
Tom
Iwouldtothankmyfamily.Devika,youhelpedmewriteafewsentences.Ihope
yourloveofwritingcontinuestogrow.Arjun,yougavemesomecreativeideasfor
differentmetrics.Iknowyourcuriositywilltakeyoufar.TomywifeMonika,thank
youforallthesupport,encouragement,andfaithyouhadinmewhilewritingthis
book and especially for your patience on all those evenings when I was writing
while you were helping thekids with their homework.
Bill
xvii
CHAPTER
1
Introduction
The goal of this book is to show how usability metrics can be a powerful tool for
successfully evaluating the user experience for any product. When some people
think about usability metrics, they feel overwhelmed by complicated formulas,
contradictory research, and advanced statistical methods. We hope to demystify
much of the research and focus on the practical application of usability metrics.
We’ll walk you through a step-by-step approach to collecting, analyzing, and
presenting usability metrics. We’ll help you choose the right metrics for each
situation or application, and show you how to use them to produce reliable,
actionable results without breaking your budget. We’ll give you guidelines and
tips for analyzing a wide range of usability metrics and provide many different
examples of how to present usability metrics to others in simple and effective
ways.
Our intention is to make this book a practical, how-to guide about measuring
theusabilityofanyproduct.Wearen’tgoingtogiveyoualotofformulas;infact,
thereareveryfew.Thestatisticswillbefairlylimited,andthecalculationscanbe
doneeasilyinExcelorsomeothercommonsoftwarepackageorwebapplication.
Ourgoalistogiveyouthetoolsyouneedtoevaluatetheusabilityofanyproduct,
withoutoverwhelming you with unnecessarydetails.
This book is both product- and technology-neutral. The usability metrics we
describe can be used for practically any type of product and any type of technol-
ogy. This is one of the great features of usability metrics: They aren’t just for
websites or any single technology. For example, task success and satisfaction are
equally valid whether you evaluate a website, a treadmill, or a toaster. More
advancedtechnologies,suchaswebsites,mobilephones,software,andconsumer
electronics, are of special concern because they’re generally more complicated,
but the basic premise remains the same.
The ‘‘half-life’’ of usability metrics is much greater than any specific design or
technology. Despite all the changes in technology, the metrics essentially stay the
same. Some metrics may change with the development of new technologies to
measure usability, but the underlying phenomena being measured don’t change.
Eye-trackingisagreatexample.Manyresearcherswantedamethodfordetermining 1
2 CHAPTER 1 Introduction
whereexactlyauserislookingonthescreen.Now,withthelatestadvancesineye-
trackingtechnology,measurementhasbecomemucheasierandfarmoreaccurate.
So why did we write this book? There’s certainly no shortage of books on
human factors, statistics, experimental design, and usability methods. Some of
those books even cover the more common usability metrics. Does a book that
focuses entirely on usability metrics even make sense? Obviously, we think so. In
our (humble) opinion, this book makes five unique contributions to the realm of
usabilitypublications:
n We take a comprehensive look at usability metrics. No other books review so
many different usability metrics. We provide details on collecting, analyzing,
and presenting nearly every type of usability metric you could possibly use.
n Thisbooktakesapracticalapproach.Weassumeyou’reinterestedinapplying
usabilitymetricsaspartofyourjob.Wedon’twasteyourtimewithunnecessary
details. We want you to be able to use these metrics easily every day. If you’re
interested in the theoretical side, we point you to additional resources.
n We provide help in making the right decisions about usability metrics. One of
themostdifficultaspectsofausabilityprofessional’sjobisdecidingwhetherto
collectmetricsand,ifso,whichonestouse.Weguideyouthroughthedecision
process so that you find the right metrics for your situation.
n Weprovidemanyexamplesofhowusabilitymetricshavebeenappliedwithin
different organizations and how they have been used to address specific
usability questions. We also provide in-depth case studies to help you deter-
mine how best to use the information revealed by the usability metrics.
n Wepresent usability metrics thatcanbeusedwithanyproduct ortechnology.
We take a broad view so that these usability metrics can be helpful throughout
your careereven as technology and products change.
1.1 ORGANIZATION OF THIS BOOK
Thisbookisorganizedintothreemainparts.Thefirstone(Chapters1–3)provides
backgroundinformationneededtogetuptospeed onusability metrics.Thispart
isintendedforthosewhoarelessfamiliarwithusability,dataanalysis,orstatistics.
n Chapter 1 provides an overview of usability and usability metrics. We define
user experience, usability, and different types of usability metrics; discuss the
valueofmeasuringtheuserexperience;anddispelsomeofthecommonmyths
aboutusability metrics.
n Chapter 2 includes background information on usability data and some basic
statistical concepts. We walk you through a step-by-step process to set up a
1.1 Organization of This Book 3
usability study using different metrics and provide a guide for performing
common statistical procedures related to different usability methods.
n Chapter 3 focuses on planning a usability study, including defining partici-
pant goals and study goals and choosing the right metrics for a wide variety of
situations.
The second part (Chapters 4–9)reviewsfive generaltypes ofusabilitymetrics,
as well as some special topics that don’t fall neatly into any single type. For each
metric,weexplainwhatitis,whentouseit,andwhennottouseit.Weshowyou
how to collect the data and different ways to analyze and present it. We provide
examples of how ithas been used in real-world usability studies.
n Chapter 4covers various types ofperformancemetrics,includingtask success,
timeontask,errors,efficiency,andeaseoflearning.Thesemetricsaregrouped
under an‘‘umbrella’’ofperformance becausetheymeasure different aspectsof
the user’s behavior.
n Chapter 5 looks at measuring usability issues. Usability issues can easily be
quantified by measuring the frequency, severity, and type of issue. We also
discusssomeofthedebatesaboutappropriatesamplesizesandhowtocapture
usability issues reliably.
n Chapter6focusesonself-reportedmetrics,suchassatisfaction,expectations,ease-
of-useratings,usefulness,andawareness.Self-reportedmetricsarebasedonwhat
users share about their experiences, not what the usability specialist measures
abouttheiractualbehaviors.
n Chapter 7 is devoted to behavioral and physiological metrics. These metrics
include eye-tracking, facial expressions, and various measures of stress. All of
thesemetricscapturesomethingabouthowthebodybehavesasaresultofthe
experience ofworkingwith auserinterface.
n Chapter8discusseshowtocombinedifferenttypesofmetricsandderivenew
metrics. Sometimes it’s helpful to get an overall assessment of the usability of
anyproduct.Thisglobalassessmentisachievedbycombiningdifferenttypesof
metrics into a single usability score, summarizing them in a usability scorecard,
or comparing them to expert performance.
n Chapter 9 presents special topics that we believe are important but that don’t
fit squarely into one of the five general categories. These include A/B testing on
alivewebsite,card-sortingdata,SixSigma,accessibilitydata,andreturnoninvest-
ment(ROI).
The third part (Chapters 10–11) shows how usability metrics are put into
practice. In this part, we highlight how usability metrics are actually used within
different types of organizations and how to promote the use of metrics within an
organization.
4 CHAPTER 1 Introduction
n Chapter 10 presents six case studies. Each case study reviews how different
typesofusabilitymetricswereused,howthedatawerecollectedandanalyzed,
and the results. These case studies were drawn from usability practitioners in
various types of organizations, including consulting, government, industry, and
not-for-profit/education.
n Chapter 11 provides ten steps to help you move forward in using metrics
withinyourorganization.Wediscusshowusabilitymetricscanfitwithindiffer-
ent types of organizations, practical tips for making metrics work within your
organization, and recipes forsuccess.
1.2 WHAT IS USABILITY?
Before we try to measure usability, we should know what it is and what it isn’t.
Therearemanydefinitionsofusability—maybeevenoneforeverypersoninvolved
in thefield! We’re going to focus on three definitions.
TheInternationalStandardsOrganization(ISO9241-11)identifiesthreeaspects
ofusability,definingitas‘‘theextenttowhichaproductcanbeusedbyspecified
userstoachievespecifiedgoalswitheffectiveness,efficiency,andsatisfactionina
specified contextof use.’’
The Usability Professionals Association (UPA) definition focuses more on the
product development process: ‘‘Usabilityis an approach to product development
thatincorporatesdirectuserfeedbackthroughoutthedevelopmentcycleinorder
to reduce costs and create products and toolsthat meetuser needs.’’
InhispopularbookDon’tMakeMeThink,SteveKrug(2000)providesasimple
perspective: ‘‘Usability really just means making sure that something works well:
thatapersonofaverage(orevenbelowaverage)abilityandexperiencecanusethe
thing—whether it’s a website, a fighter jet, or a revolving door—for its intended
purpose without getting hopelessly frustrated.’’
Allthreeofthesedefinitions,aswellasotherdefinitionsofusability,sharesome
commonthemes:
n A user isinvolved.
n Thatuser isdoing something.
n Thatuser isdoing somethingwith a product, system, orother thing.
Some people distinguish between the terms usability and user experience.
Usability is usually considered the ability of the user to use the thing to carry
out a task successfully, whereas user experience takes a broader view, looking at
theindividual’sentireinteractionwiththething,aswellasthethoughts,feelings,
and perceptions that result from that interaction. We take a very broad view of
usability and examine the entire user experience. Therefore, when we talk about
‘‘measuring usability,’’we’re reallylooking at theentire user experience.