Table Of ContentSpringer Tracts in Advanced Robotics 134
Adrià Colomé
Carme Torras
Reinforcement
Learning of
Bimanual Robot
Skills
Springer Tracts in Advanced Robotics
Volume 134
Series Editors
Bruno Siciliano, Dipartimento di Ingegneria Elettrica e Tecnologie
dell’Informazione, Università degli Studi di Napoli Federico II, Napoli, Italy
Oussama Khatib, Artificial Intelligence Laboratory, Department of Computer
Science, Stanford University, Stanford, CA, USA
Advisory Editors
NancyAmato,ComputerScience&Engineering,TexasA&MUniversity,College
Station, TX, USA
Oliver Brock, Fakultät IV, TU Berlin, Berlin, Germany
Herman Bruyninckx, KU Leuven, Heverlee, Belgium
WolframBurgard,InstituteofComputerScience,UniversityofFreiburg,Freiburg,
Baden-Württemberg, Germany
Raja Chatila, ISIR, Paris cedex 05, France
Francois Chaumette, IRISA/INRIA, Rennes, Ardennes, France
Wan Kyun Chung, Robotics Laboratory, Mechanical Engineering, POSTECH,
Pohang, Korea (Republic of)
Peter Corke, Science and Engineering Faculty, Queensland University of
Technology, Brisbane, QLD, Australia
Paolo Dario, LEM, Scuola Superiore Sant’Anna, Pisa, Italy
Alessandro De Luca, DIAGAR, Sapienza Università di Roma, Roma, Italy
Rüdiger Dillmann, Humanoids and Intelligence Systems Lab, KIT - Karlsruher
Institut für Technologie, Karlsruhe, Germany
Ken Goldberg, University of California, Berkeley, CA, USA
John Hollerbach, School of Computing, University of Utah, Salt Lake, UT, USA
LydiaE.Kavraki,DepartmentofComputerScience,RiceUniversity,Houston,TX,
USA
Vijay Kumar, School of Engineering and Applied Mechanics, University of
Pennsylvania, Philadelphia, PA, USA
Bradley J. Nelson, Institute of Robotics and Intelligent Systems, ETH Zurich,
Zürich, Switzerland
Frank Chongwoo Park, Mechanical Engineering Department, Seoul National
University, Seoul, Korea (Republic of)
S. E. Salcudean, The University of British Columbia, Vancouver, BC, Canada
Roland Siegwart, LEE J205, ETH Zürich, Institute of Robotics & Autonomous
Systems Lab, Zürich, Switzerland
Gaurav S. Sukhatme, Department of Computer Science, University of Southern
California, Los Angeles, CA, USA
TheSpringerTractsinAdvancedRobotics(STAR)publishnewdevelopmentsand
advances in the fields of robotics research, rapidly and informally but with a high
quality. The intent is to cover all the technical contents, applications, and
multidisciplinary aspects of robotics, embedded in the fields of Mechanical
Engineering,ComputerScience,ElectricalEngineering,Mechatronics,Control,and
Life Sciences, as well as the methodologies behind them. Within the scope of the
series are monographs, lecture notes, selected contributions from specialized
conferences and workshops, as well as selected PhD theses.
Special offer: For all clients with a print standing order we offer free access to the
electronic volumes of the Series published in the current year.
Indexed by DBLP, Compendex, EI-Compendex, SCOPUS, Zentralblatt Math,
Ulrich’s, MathSciNet, Current Mathematical Publications, Mathematical Reviews,
MetaPress and Springerlink.
More information about this series at http://www.springer.com/series/5208
à é
Adri Colom Carme Torras
(cid:129)
Reinforcement Learning
of Bimanual Robot Skills
123
AdriàColomé CarmeTorras
Institut deRobòticai Informàtica Industrial Institut deRobòticai Informàtica Industrial
(UPC-CSIC) (UPC-CSIC)
Barcelona,Spain Barcelona,Spain
ISSN 1610-7438 ISSN 1610-742X (electronic)
SpringerTracts inAdvanced Robotics
ISBN978-3-030-26325-6 ISBN978-3-030-26326-3 (eBook)
https://doi.org/10.1007/978-3-030-26326-3
©SpringerNatureSwitzerlandAG2020
Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpart
of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations,
recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission
orinformationstorageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilar
methodologynowknownorhereafterdeveloped.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this
publicationdoesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfrom
therelevantprotectivelawsandregulationsandthereforefreeforgeneraluse.
The publisher, the authors and the editors are safe to assume that the advice and information in this
book are believed to be true and accurate at the date of publication. Neither the publisher nor the
authors or the editors give a warranty, expressed or implied, with respect to the material contained
hereinorforanyerrorsoromissionsthatmayhavebeenmade.Thepublisherremainsneutralwithregard
tojurisdictionalclaimsinpublishedmapsandinstitutionalaffiliations.
ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSwitzerlandAG
Theregisteredcompanyaddressis:Gewerbestrasse11,6330Cham,Switzerland
To Raquel and Ivet
Foreword
Robotics is undergoing a major transformation in scope and dimension. From a
largely dominant industrial focus, robotics is rapidly expanding into human envi-
ronmentsandvigorouslyengagedinitsnewchallenges.Interactingwith,assisting,
serving, and exploring with humans, the emerging robots will increasingly touch
people and their lives.
Beyond its impact on physical robots, the body of knowledge robotics has
produced is revealing a much wider range of applications reaching across diverse
research areas and scientific disciplines, such as biomechanics, haptics, neuro-
sciences,virtualsimulation,animation,surgery,andsensornetworksamongothers.
Inreturn,thechallengesofthenewemergingareasareprovinganabundantsource
ofstimulationandinsightsforthefieldofrobotics.Itisindeedattheintersectionof
disciplines that the most striking advances happen.
TheSpringer TractsinAdvancedRobotics(STAR)isdevotedtobringingtothe
research community the latest advances in the robotics field on the basis of their
significance and quality. Through a wide and timely dissemination of critical
researchdevelopmentsinrobotics,ourobjectivewiththisseriesistopromotemore
exchanges and collaborations among the researchers in the community and con-
tribute to further advancements in this rapidly growing field.
ThemonographbyAdriàColoméandCarmeTorrasisbasedonthefirstauthor’s
doctoral thesis. It conjugates the two research worlds of artificial intelligence and
robotics in one original work on reinforcement learning for bimanual robots. The
contentsareeffectivelyorganizedintwoparts:(i)compliantredundantrobotcontrol,
which is based on the well-known Jacobian-based CLIK algorithm for solving
inversekinematics,alongwithanestimateoftheexternalforceandfrictionmodel;
(ii) reinforcement learning with movement primitives, which is based on a new
Dual REPS algorithm for fast learning of robot motion, along with a reduction of
dimensionality of themotion parametrisation.
vii
viii Foreword
Richofexamplesdevelopedonasystemoftwoanthropomorphicarmscarrying
outpracticalbimanualtasks,thisvolumewasafinalistforthe2018GeorgesGiralt
Ph.D. Award for the best doctoral thesis in Europe. A very fine addition to the
STAR series!
Naples, Italy Bruno Siciliano
June 2019 STAR Editor
Preface
In our culture, robots have been in novels and cinema for a long time, but it has
beenspeciallyinthelasttwodecadeswhentheimprovementsinhardware—better
computational power and components—and advances in Artificial Intelligence
(AI), have allowed robots to start sharing spaces with humans. Such situations
require, aside from ethical considerations, robots to be able to move with both
compliance and precision, and learn at different levels, such as perception, plan-
ning, and motion, being the latter the focus of this work.
The first issue addressed in this monograph is inverse kinematics for redundant
robot manipulators, i.e., positioning the robot joints so as to reach a certain
end-effector pose. We opt for iterative solutions based on the inversion of the
kinematic Jacobian of a robot, and propose to filter and limit the gains in the
spectraldomain,whilealsounifyingsuchapproachwithacontinuous,multipriority
scheme. Such inverse kinematics method is then used to derive manipulability in
thewholeworkspaceofananthropomorphicarm,andthecoordinationoftwoarms
is subsequently optimized by finding their best relative positioning.
Having solved the kinematic issues, a robot learning within a human environ-
mentneedstomovecompliantly,withlimitedamountofforce,inordernottoharm
anyhumansorcauseanydamage,whilebeingaspreciseaspossible.Therefore,we
developed two dynamic models for the same redundant arm we had analysed
kinematically:ThefirstisbasedonlocalmodelswithGaussianprojections,andthe
second characterizing the most problematic term of the dynamics, namely friction.
Such models allowed us to implement feed-forward controllers, where we can
actively change the weights in the compliance-precision trade-off. Moreover, we
used such models to predict external forces acting on the robot, without the use of
force sensors.
Afterwards, we noticed that bimanual robots must coordinate their components
(or limbs)andbeabletoadapttonewsituations withease.Overthelastdecade, a
number of successful applications for learning robot motion tasks have been pub-
lished. However, due to the complexity of a complete system including all the
required elements, most of these applications involve only simple robots with a
large number of high-end technology sensors, or consist of very simple and
ix
x Preface
controlled tasks. Using our previous framework for kinematics and control, we
relied on two types of movement primitives to encapsulate robot motion. Such
movement primitives are very suitable for using reinforcement learning. In par-
ticular, we applied direct policy search, which uses the motion parametrization as
the policy itself.
Inordertoincreasethelearningspeedinrealrobotapplications,wegeneralized
apolicysearchalgorithmtogivesomeimportancetosamplesyieldingabadresult,
and we paid special attention to the dimensionality of the motion parametrization.
We reduced such dimensionality with linear methods, using the rewards obtained
throughmotion repetition andexecution.Wetestedsuchframeworkina bimanual
task performed by two anthropomorphic arms, such as the folding of garments,
showing how a reduced dimensionality can provide qualitative information about
robot couplings and help to speed up the learning of tasks when robot motion
executions are costly.
ThisworkhasbeensupportedbytheSpanishMinistryofEducation,Cultureand
Sport via an FPU doctoral grant AP2010-1989, and by the European research
projects IntellAct (FP7-ICT-2009-6-269959) and I-DRESS (PCIN-2015-147), the
Spanish national project RobInstruct (TIN2014-58178-R), and the European
Research Council Advanced Grant CLOTHILDE (ERC-2016-ADG-741930).
Barcelona, Spain Adrià Colomé
May 2019 Carme Torras
