Emotions and Affect in Human Factors and Human-Computer Interaction

591 Pages·2017·19.48 MB·English

by  Myounghoon Jeon (Auth.)

EMOTIONS AND
AFFECT IN HUMAN
FACTORS AND
HUMAN–COMPUTER
INTERACTION
Edited by
Myounghoon Jeon
Michigan Technological University, Houghton, MI, United States
Copyright
Academic Press is an imprint of Elsevier
125 London Wall, London EC2Y 5AS, United Kingdom
525 B Street, Suite 1800, San Diego, CA 92101-4495, United States
50 Hampshire Street, 5th Floor, Cambridge, MA 02139, United States
The Boulevard, Langford Lane, Kidlington, Oxford OX5 1GB, United Kingdom
Copyright © 2017 Elsevier Inc. All rights reserved.
No part of this publication may be reproduced or transmitted in any form or by any means,
electronic or mechanical, including photocopying, recording, or any information storage
and retrieval system, without permission in writing from the publisher. Details on how to
seek permission, further information about the Publisher’s permissions policies and our
arrangements with organizations such as the Copyright Clearance Center and the Copyright
Licensing Agency, can be found at our website: www.elsevier.com/permissions.
This book and the individual contributions contained in it are protected under copyright by
the Publisher (other than as may be noted herein).
Notices
Knowledge and best practice in this field are constantly changing. As new research and
experience broaden our understanding, changes in research methods, professional practices,
or medical treatment may become necessary.
Practitioners and researchers must always rely on their own experience and knowledge
in evaluating and using any information, methods, compounds, or experiments described
herein. In using such information or methods they should be mindful of their own safety
and the safety of others, including parties for whom they have a professional responsibility.
To the fullest extent of the law, neither the Publisher nor the authors, contributors, or editors,
assume any liability for any injury and/or damage to persons or property as a matter of
products liability, negligence or otherwise, or from any use or operation of any methods,
products, instructions, or ideas contained in the material herein.
Library of Congress Cataloging-in-Publication Data
A catalog record for this book is available from the Library of Congress
British Library Cataloguing-in-Publication Data
A catalogue record for this book is available from the British Library
ISBN: 978-0-12-801851-4
For information on all Academic Press publications visit our website at
https://www.elsevier.com/books-and-journals
Publisher: Nikki Levy
Acquisition Editor: Emily Ekle
Editorial Project Manager: Barbara Makinster
Production Project Manager: Nicky Carter
Designer: Matthew Limbert
Typeset by Thomson Digital
List of Contributors
Jacob Aday Northern Michigan University, Marquette, MI, United States
Robert K. Atkinson Arizona State University, Tempe, AZ, United States
Mustafa Baydogan Bogazici University, Istanbul, Turkey
Anthony F. Beavers The University of Evansville, Evansville, IN, United States
Jenay M. Beer University of South Carolina, Columbia, SC, United States
Tibor Bosse VU University Amsterdam, Amsterdam, The Netherlands
Winslow Burleson New York University, New York, NY, United States
Rafael A. Calvo School of Electrical and Information Engineering, University of
Sydney, Sydney, NSW, Australia
Joshua M. Carlson Northern Michigan University, Marquette, MI, United States
Andrea Cavallaro Queen Mary University of London, London, United Kingdom
Maria Elena Chavez-Echeagaray Arizona State University, Tempe, AZ, United
States
David Cherry Clemson University, Clemson, SC, United States
Shaundra B. Daily University of Florida, Gainesville, FL, United States
Shelby S. Darnell Clemson University, Clemson, SC, United States
Oya Demirbilek University of New South Wales, Sydney, NSW, Australia
Stephen H. Fairclough Liverpool John Moores University, Liverpool, United
Kingdom
Paul A. Fishwick University of Texas at Dallas, Richardson, TX, United States
Joseph P. Forgas University of New South Wales, Sydney, NSW, Australia
Andrea Gaggioli Catholic University of Sacred Heart; Istituto Auxologico
Italiano, Milan, Italy
Javier Gonzalez-Sanchez Arizona State University, Tempe, AZ, United States
Hatice Gunes University of Cambridge, Cambridge, United Kingdom
Peter A. Hancock University of Central Florida, Orlando, FL, United States
Bruce Hanington Carnegie Mellon University, Pittsburgh, PA, United States
Eva Hudlicka Psychometrix Associates, College of Information and Computer
Sciences, University of Massachusetts-Amherst, Amherst, MA, United States
Joseph Isaac University of Florida, Gainesville, FL, United States
Melva T. James Clemson University, Clemson, SC; Massachusetts Institute of
Technology, Lexington, MA, United States
Myounghoon Jeon Michigan Technological University, Houghton, MI, United
States
xiii
xiv List of Contributors
Karina R. Liles University of South Carolina, Columbia, SC, United States
Seyedeh Maryam Fakhrhosseini Michigan Technological University, Houghton,
MI, United States
Tal Oron-Gilad Ben-Gurion University of the Negev, Be’er Sheva, Israel
Sujan Pakala University of South Carolina, Columbia, SC, United States
Dorian Peters School of Electrical and Information Engineering, University of
Sydney, Sydney, NSW, Australia
John J. Porter, III Clemson University, Clemson, SC, United States
Andreas Riener Johannes Kepler University Linz, Linz, Austria; Ingolstadt
University of Applied Sciences, Ingolstadt, Germany
Giuseppe Riva Catholic University of Sacred Heart; Istituto Auxologico Italiano,
Milan, Italy
Will Rizer Northern Michigan University, Marquette, MI, United States
Tania Roy Clemson University, Clemson, SC, United States
James A. Russell Boston College, Chestnut Hill, MA, United States
Evangelos Sariyanidi Queen Mary University of London, London, United
Kingdom
Justin P. Slattery Indiana University, Bloomington, IN, United States
Xian Wu University of South Carolina, Columbia, SC, United States
Toshimasa Yamanaka University of Tsukuba, Tsukuba, Japan
Preface
THE ORGANIZATION OF THE HANDBOOK
This handbook aims to serve as a “field guide” in conducting affect-
related research and design projects in H/F and HCI, by introducing
necessary concepts, approaches, methods, applications, and emerging ar-
eas so that readers can apply those ingredients to their own area. On the
other hand, since this handbook deals with comprehensive topics, I hope
that experts in this domain can also utilize this handbook as a necessary
reference or a teaching material. The handbook is organized in the
following way.
PART I: FOUNDATIONS OF AFFECTIVE SCIENCES
Part I provides a theoretical background of Affective Sciences. More and
more research on emotions and affect has recently appeared in applied H/F
and HCI settings, whereas considerable research has not been based on
affect theory or mechanism. Engineers and designers can learn and apply
these psychological theories and mechanisms to account for their affect-
related research and can develop their own domain-specific theory. In
Chapter 1 Jeon presents an overview of Affective Sciences in H/F and HCI.
This is a compact version of the handbook. In Chapter 2 Aday, Rizer, and
Carlson introduce the neural mechanisms associated with affective phe-
nomena, such as appraisal, reactivity, feeling, and regulation with a focus
on the role of amygdala. Then, Forgas provides an extensive review on the
affective effects on cognitive processes—content of thinking and quality
of information processing with recent empirical studies (Chapter 3). In
Chapter 4 Russell proposes a new conceptual framework of affect about
commonalities and differences among individuals and cultures, high-
lighting subjective experience and facial expressions of emotion. Beavers
and Slattery discuss a moral dilemma and ethical issues, which arise,
specifically from “simulated affect” of machines in affective computing.
This chapter should serve as a good starting point about other ethical
issues regarding affect induction, detection, data storage and access, and
intervention for affect regulation, which are critical components in affect-
related projects in H/F and HCI.
xv
xvi Preface
PART II: FRAMEWORKS OF AFFECTIVE SCIENCES
IN HUMAN FACTORS AND HCI
In Part II, representative (most widely cited, discussed, and currently
followed) approaches to Affective Sciences in H/F and HCI are intro-
duced, including Emotional Design, Hedonomics, Kansei Engineering,
and Affective Computing. Chapters cover each approach’s background,
goal, development, characteristics, applications, and future directions.
Readers will be able to learn how these approaches are similar to each
other and how they are distinct from each other. Based on that, readers
can get some sense of what type of approach they should take for their
research domain. This part begins with Hanington’s Chapter 6 about de-
sign and emotion; its history, theories, methodologies, and approaches.
This chapter also supplies a bird’s eye view about this part. Next, Oron-
Gilad and Hancock reveal the evolution of human factors from traditional
Ergonomics into Hedonomics in Chapter 7. They emphasize to catch up
the tremendous progress in the domain and change their focus from func-
tional usability to hedonomic fulfillment. In Chapter 8 Yamanaka tells us
about philosophical background of the Kansei approach and relationships
among Kansei, Kansei science, and Kansei Engineering. He finishes up his
chapter by demonstrating a design example which Kansei design proce-
dure has been applied for. Daily and her coworkers wrap up the history,
core components (i.e., affect sensing and generation), current applications,
and future trends of Affective Computing. They also include interesting
topics on ethical consideration.
PART III: METHODOLOGIES: INTRODUCTION
AND EVALUATION OF TECHNIQUES
To conduct affect research, systematic knowledge about practical
methods is necessary. Part III introduces ample methodologies, such as
affect induction, measurement, and detection techniques. Also, regula-
tion strategies and computational models for training and intervention
system development are included. Readers can learn the pros and cons
of each method and how to choose or combine those methods for their
own research. Both Chapters 10 and 11 are written to offer a boot camp for
novice researchers so that they are able to follow the steps. In Chapter 10,
FakhrHosseini and Jeon demonstrate diverse affect induction methods
used in actual research settings—listening to music, writing experiences,
reading passages, watching videos, or photos, etc. Then, Gonzalez-
Sanchez and his coworkers describe sensing technologies (brain–computer
interfaces, facial expression detection, eye-tracking systems, physiologi-
cal sensors, body language recognition, and writing patterns) and detail
Preface xvii
their technical know-how from data collection to analysis in Chapter 11.
In a similar line, Sariyanidi, Gunes, and Cavallaro discuss facial expres-
sion more in depth in Chapter 12. Specifically, they focus on registration
and representation, which are important both in cognitive sciences and
computer science. Even without understanding the algorithms, readers
can readily follow their ideas and get the gist. This part closes with com-
putational models of affect regulation by Bosse (Chapter 13). He covers
emotion modeling and its application to regulation processes, which can
be used in a variety of application settings.
PART IV: APPLICATIONS: CASE STUDIES
AND APPLIED EXAMPLES
Application domains are selected based on the amount of existing af-
fect research and documented literature in a specific domain. However,
these are just a few selected areas, not exhaustive. Chapters in this part
discuss various H/F and HCI application areas, including product de-
sign, human–robot interaction, healthcare, and driving. Core research
topics, design practices and case studies, theoretical and practical issues,
and design guidelines are discussed. Demirbilek in Chapter 14 provides
abundant product design examples, which include emotional sensory mo-
dalities—look (color), feel (touch), and sound. She also delineates positive
emotional triggers for delightful experiences, such as fun, cuteness, and
familiarity. In Chapter 15 Beer and her coworkers give an overview about
affective human–robot interaction. They define both humans and robots
as social entities and explore the existing applications and future possi-
bilities of social robots for aging, companionship, and education. Then, in
Chapter 16 Hudlicka discusses the state-of-the-art in modeling emotion–
cognition interaction and the application of these models for understand-
ing the mechanisms mediating psychopathology and therapeutic action,
and for facilitating more accurate assessment and diagnosis of affective
disorders. Changing gear, Jeon shows lacking of the driving models that
have emotional components and presents recent empirical research out-
comes about the effects of various affective states on driving in Chapter 17.
PART V: EMERGING AREAS
The last part briefly touches emerging interdisciplinary areas related
to Affective Sciences in H/F and HCI, including Positive Technology,
Subliminal Perception, Physiological Computing, and Aesthetic Com-
puting. There are new conferences, workshops, journal special issues, or
new books for these emerging areas. The presence of all these relevant
xviii Preface
disciplines supports the importance of affective approaches to H/F and
HCI. Chapters in this part describe their background, goal, research topics,
and how they are relevant to Affective Sciences. This part provides a new
perspective to understand a big picture of Affective Sciences. Gaggioli and
coworkers start the part with Chapter 18, by depicting how positive psy-
chology has sought happiness and well-being, which has brought about
positive technology and positive computing. They offer theoretical back-
ground, applications, and future research agenda of these approaches to
human happiness and well-being. In Chapter 19 Riener questions if we
can provide additional information to users while they are not aware of
that perception, which is called, “subliminal perception.” He shows com-
pelling examples and potentials of subliminal approaches for different
sensory channels, specifically in the automotive domain. In Chapter 20
Fairclough provides an overview about physiological computing in which
physiological data from the brain and body are transformed into input
control to inform software adaptation. He explains how a dynamic user
representation can be used as a form of input control and guides a process
of intelligent adaptation and this adaptation can enhance human–com-
puter interaction ultimately. In the last chapter of this handbook, Fishwick
answers to core questions in aesthetic computing: “How can embodied
cognition be situated within formal languages?” and “How can embodied
cognition result in novel computer interfaces for formal languages?” He
challenges us to build a new generation of human–computer interfaces
that are informed by embodied principles and use these principles as de-
sign elements for interacting with formal languages.
As readers can see, I tried to offer multiple perspectives on the topic for
a holistic approach to Affective Sciences in H/F and HCI. It was a long but
rewarding journey to edit this handbook. I have taught the course with
the same title as this handbook, but I myself have learned a lot more about
the domain while editing this handbook. By no means, this handbook ad-
dresses all relevant topics of Affective Sciences in H/F and HCI. However,
this volume should help frame useful, timely questions about the roles of
emotions and affect in interactions between people and technology. Yes, it
cannot answer them just yet. In the next turn, this book could be further
updated based on more empirical research studies. The approach outlined
in this handbook is expected to begin to close the existing gap between the
traditional affect research and the emerging field of affective design and
affective computing.
I hope that readers in this topic—scholars, students, designers, engi-
neers, and practitioners, can benefit from reading diverse perspectives
and applications, and ultimately integrating multiple approaches into
their own work.
Myounghoon Jeon
Acknowledgments
This handbook would not have been possible without the contribution
of all the authors, who have generously shared their invaluable views and
research outcomes to make this book. I deeply appreciate their time and
effort for their excellent work.
I would like to thank Emily Ekle, Timothy Bennett, Barbara Makinster,
and Nicky Carter at Elsevier for suggesting the edits in this handbook
and managing the publication processes. I also thank Dr. Bruce Seely,
Dr. Susan Amato-Henderson, and Dr. Min Song for their practical advice
and strong support. Finally, I am extremely grateful to my academic advi-
sors—Dr. Kwang Hee Han and Dr. Bruce Walker, my students in the Mind
Music Machine Lab, and my family.
xix
C H A P T E R
1
Emotions and Affect in Human
Factors and Human–Computer
Interaction: Taxonomy,
Theories, Approaches,
and Methods
Myounghoon Jeon
Michigan Technological University, Houghton, MI, United States
INTRODUCTION
Emotions and affect are known as a motivating and guiding force in
our attention and perception (Izard, 1993). Psychologists even accept
that it is impossible for people to have a thought or perform an action
without engaging their emotional system, at least unconsciously (Nass
et al., 2005). However, despite their importance and prevalence in every-
day life, emotions and affect have had fewer chances to be a dominant
topic of human factors (HF) and human–computer interaction (HCI) for a
long time. Given that the information processing approach of traditional
Cognitive Sciences has been a dominant paradigm of HF and HCI, there
has not been much room for emotions and affect in the mainstream of
human–machine system research until recently. However, affective ele-
ments allow for a systems approach (Czaja and Nair, 2006) and a more
holistic view to understanding the human–machine system. For example,
affective states have been known to play a major role in influencing all
aspects of situation assessment and belief information, from cue identi-
fication and extraction to situation classification, and decision selection
Emotions and Affect in Human Factors and Human–Computer Interaction. http://dx.doi.org/10.1016/B978-0-12-801851-4.00001-X
Copyright © 2017 Elsevier Inc. All rights reserved.
3