Deep Learning in Natural Language Processing

338 Pages·2018·5.61 MB·English

by  Li Deng

Editors
Li Deng · Yang Liu
Deep Learning
in Natural
Language
Processing
Deep Learning in Natural Language Processing
Li Deng Yang Liu
(cid:129)
Editors
Deep Learning in Natural
Language Processing
123
Editors
LiDeng Yang Liu
AIResearch at Citadel TsinghuaUniversity
Chicago, IL Beijing
USA China
and
AIResearch at Citadel
Seattle, WA
USA
ISBN978-981-10-5208-8 ISBN978-981-10-5209-5 (eBook)
https://doi.org/10.1007/978-981-10-5209-5
LibraryofCongressControlNumber:2018934459
©SpringerNatureSingaporePteLtd.2018
Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpart
of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations,
recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission
orinformationstorageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilar
methodologynowknownorhereafterdeveloped.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this
publicationdoesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfrom
therelevantprotectivelawsandregulationsandthereforefreeforgeneraluse.
The publisher, the authors and the editors are safe to assume that the advice and information in this
book are believed to be true and accurate at the date of publication. Neither the publisher nor the
authorsortheeditorsgiveawarranty,expressorimplied,withrespecttothematerialcontainedhereinor
for any errors or omissions that may have been made. The publisher remains neutral with regard to
jurisdictionalclaimsinpublishedmapsandinstitutionalaffiliations.
Printedonacid-freepaper
ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSingaporePteLtd.
partofSpringerNature
Theregisteredcompanyaddressis:152BeachRoad,#21-01/04GatewayEast,Singapore189721,
Singapore
Foreword
“Writtenbyagroupofthemostactiveresearchersinthefield,ledbyDr.Deng,an
internationallyrespectedexpertinboth NLPanddeeplearning,thisbookprovides
acomprehensiveintroductiontoandup-to-datereviewofthestateofartinapplying
deep learning to solve fundamental problems in NLP. Further, the book is highly
timely, as demands for high-quality and up-to-date textbooks and research refer-
enceshaverisendramaticallyinresponsetothetremendousstridesindeeplearning
applications to NLP. The book offers a unique reference guide for practitioners in
various sectors, especially the Internet and AI start-ups, where NLP technologies
are becoming an essential enabler and a core differentiator.”
Hongjiang Zhang (Founder, Sourcecode Capital; former CEO of KingSoft)
“This book provides a comprehensive introduction to the latest advances in deep
learning applied to NLP. Written by experienced and aspiring deep learning and
NLP researchers, it covers a broad range of major NLP applications, including
spoken language understanding, dialog systems, lexical analysis, parsing, knowl-
edgegraph,machinetranslation,questionanswering,sentimentanalysis,andsocial
computing.
The book is clearly structured and moves from major research trends, to the
latest deep learning approaches, to their limitations and promising future work.
Given its self-contained content, sophisticated algorithms, and detailed use cases,
the book offers a valuable guide for all readers who are working on or learning
about deep learning and NLP.”
Haifeng Wang (Vice President and Head of Research, Baidu; former President
of ACL)
“In2011,atthedawnofdeeplearninginindustry,Iestimated thatinmostspeech
recognitionapplications,computersstillmade5to10timesmoreerrorsthanhuman
subjects, and highlighted the importance of knowledge engineering in future
directions.Withinonlyahandfulofyearssince,deeplearninghasnearlyclosedthe
gap in the accuracy of conversational speech recognition between human and
computers. Edited and written by Dr. Li Deng—a pioneer in the recent speech
v
vi Foreword
recognitionrevolutionusingdeeplearning—andhiscolleagues,thisbookelegantly
describes this part of the fascinating history of speech recognition as an important
subfield of natural language processing (NLP). Further, the book expands this
historical perspective from speech recognition to more general areas of NLP,
offering a truly valuable guide for the future development of NLP.
Importantly,thebookputsforwardathesisthatthecurrentdeeplearningtrendis
arevolutionfromthepreviousdata-driven(shallow)machinelearningera,although
ostensibly deep learning appears to be merely exploiting more data, more com-
putingpower,andmore complex models.Indeed, asthebookcorrectlypointsout,
the current state of the art of deep learning technology developed for NLP appli-
cations, despite being highly successful in solving individual NLP tasks, has not
taken full advantage of rich world knowledge or human cognitive capabilities.
Therefore,Ifullyembracetheviewexpressedbythebook’seditorsandauthorsthat
moreadvanceddeeplearningthatseamlesslyintegratesknowledgeengineeringwill
pave the way for the next revolution in NLP.
IhighlyrecommendspeechandNLPresearchers,engineers,andstudentstoread
thisoutstandingandtimelybook,notonlytolearnaboutthestateoftheartinNLP
and deep learning, but also to gain vital insights into what the future of the NLP
field will hold.”
Sadaoki Furui (President, Toyota Technological Institute at Chicago)
Preface
Natural language processing (NLP), which aims to enable computers to process
human languages intelligently, is an important interdisciplinary field crossing
artificialintelligence,computingscience,cognitivescience,informationprocessing,
and linguistics. Concerned with interactions between computers and human lan-
guages, NLP applications such as speech recognition, dialog systems, information
retrieval, question answering, and machine translation have started to reshape the
way people identify, obtain, and make use of information.
The development of NLP can be described in terms of three major waves:
rationalism,empiricism,anddeeplearning.Inthefirstwave,rationalistapproaches
advocated the design of handcrafted rules to incorporate knowledge into NLP
systemsbasedontheassumptionthatknowledgeoflanguageinthehumanmindis
fixed in advance by generic inheritance. In the second wave, empirical approaches
assumethatrichsensoryinputandtheobservablelanguagedatainsurfaceformare
required and sufficient to enable the mind to learn the detailed structure of natural
language. As a result, probabilistic models were developed to discover the regu-
larities of languages from large corpora. In the third wave, deep learning exploits
hierarchical models of nonlinear processing, inspired by biological neural systems
to learn intrinsic representations from language data, in ways that aim to simulate
human cognitive abilities.
Theintersectionofdeeplearningandnaturallanguageprocessinghasresultedin
striking successes in practical tasks. Speech recognition is the first industrial NLP
application that deep learning has strongly impacted. With the availability of
large-scale training data, deep neural networks achieved dramatically lower
recognition errors than the traditional empirical approaches. Another prominent
successful application of deep learning in NLP is machine translation. End-to-end
neural machine translation that models the mapping between human languages
using neural networks has proven to improve translation quality substantially.
Therefore, neural machine translation has quickly become the new de facto tech-
nologyinmajorcommercialonlinetranslationservicesofferedbylargetechnology
companies: Google, Microsoft, Facebook, Baidu, and more. Many other areas of
NLP, including language understanding and dialog, lexical analysis and parsing,
vii
viii Preface
knowledge graph, information retrieval, question answering from text, social
computing, language generation, and text sentiment analysis, have also seen much
significant progress using deep learning, riding on the third wave of
NLP. Nowadays, deep learning is a dominating method applied to practically all
NLP tasks.
The main goal of this book is to provide a comprehensive survey on the recent
advances in deep learning applied to NLP. The book presents state of the art of
NLP-centricdeeplearningresearch,andfocusesontheroleofdeeplearningplayed
in major NLP applications including spoken language understanding, dialog sys-
tems, lexical analysis, parsing, knowledge graph, machine translation, question
answering, sentiment analysis, social computing, and natural language generation
(from images). This book is suitable for readers with a technical background in
computation,includinggraduatestudents,post-doctoralresearchers,educators,and
industrial researchers and anyone interested in getting up to speed with the latest
techniques of deep learning associated with NLP.
The book is organized into eleven chapters as follows:
(cid:129) Chapter 1: A Joint Introduction to Natural Language Processing and to Deep
Learning (Li Deng and Yang Liu)
(cid:129) Chapter2:DeepLearninginConversationalLanguageUnderstanding(Gokhan
Tur, Asli Celikyilmaz, Xiaodong He, Dilek Hakkani-Tür, and Li Deng)
(cid:129) Chapter 3: Deep Learning in Spoken and Text-Based Dialog Systems
(AsliCelikyilmaz,LiDeng,andDilekHakkani-Tür)
(cid:129) Chapter4:DeepLearning inLexicalAnalysisandParsing(Wanxiang Che and
Yue Zhang)
(cid:129) Chapter5:DeepLearninginKnowledgeGraph(ZhiyuanLiuandXianpeiHan)
(cid:129) Chapter6:DeepLearninginMachineTranslation(YangLiuandJiajunZhang)
(cid:129) Chapter7:DeepLearninginQuestionAnswering(KangLiuandYansongFeng)
(cid:129) Chapter 8: Deep Learning in Sentiment Analysis (Duyu Tang and Meishan
Zhang)
(cid:129) Chapter 9: Deep Learning in Social Computing (Xin Zhao and Chenliang Li)
(cid:129) Chapter 10: Deep Learning in Natural Language Generation from Images
(Xiaodong He and Li Deng)
(cid:129) Chapter 11: Epilogue (Li Deng and Yang Liu)
Chapter 1 first reviews the basics of NLP as well as the main scope of NLP
coveredinthefollowingchaptersofthebook,andthengoesinsomedepthintothe
historical development of NLP summarized as three waves and future directions.
Subsequently, in Chaps. 2–10, an in-depth survey on the recent advances in deep
learning applied to NLP is organized into nine separate chapters, each covering a
largely independent application area of NLP. The main body of each chapter is
written by leading researchers and experts actively working in the respective field.
Theoriginofthisbookwasthesetofcomprehensivetutorialsgivenatthe15th
China National Conference on Computational Linguistics (CCL 2016) held in
October 2016 in Yantai, Shandong, China, where both of us, editors of this book,
Preface ix
were active participants and were taking leading roles. We thank our Springer’s
senior editor, Dr. Celine Lanlan Chang, who kindly invited us to create this book
and who has been providing much of timely assistance needed to complete this
book. We are grateful also to Springer’s Assistant Editor, Jane Li, for offering
invaluable help through various stages of manuscript preparation.
We thank all authors of Chaps. 2–10 who devoted their valuable time carefully
preparing the content of their chapters: Gokhan Tur, Asli Celikyilmaz, Dilek
Hakkani-Tur,WanxiangChe,YueZhang,XianpeiHan,ZhiyuanLiu,JiajunZhang,
Kang Liu, Yansong Feng, Duyu Tang, Meishan Zhang, Xin Zhao, Chenliang Li,
andXiaodongHe.TheauthorsofChaps.4–9areCCL2016tutorialspeakers.They
spent a considerable amount of time in updating their tutorial material with the
latest advances in the field since October 2016.
Further,we thank numerous reviewersandreaders,Sadaoki Furui, AndrewNg,
FredJuang,KenChurch,HaifengWang,andHongjiangZhang,whonotonlygave
us much needed encouragements but also offered many constructive comments
which substantially improved earlier drafts of the book.
Finally,wegiveourappreciationstoourorganizations,MicrosoftResearchand
Citadel (for Li Deng) and Tsinghua University (for Yang Liu), who provided
excellentenvironments,supports, andencouragementsthathavebeeninstrumental
for us to complete this book. Yang Liu is also supported by National Natural
Science Foundation of China (No.61522204, No.61432013, and No.61331013).
Seattle, USA Li Deng
Beijing, China Yang Liu
October 2017
Contents
1 A Joint Introduction to Natural Language Processing and to
Deep Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Li Deng and Yang Liu
2 Deep Learning in Conversational Language Understanding . . . . . . 23
Gokhan Tur, Asli Celikyilmaz, Xiaodong He, Dilek Hakkani-Tür
and Li Deng
3 Deep Learning in Spoken and Text-Based Dialog Systems . . . . . . 49
Asli Celikyilmaz, Li Deng and Dilek Hakkani-Tür
4 Deep Learning in Lexical Analysis and Parsing . . . . . . . . . . . . . . . 79
Wanxiang Che and Yue Zhang
5 Deep Learning in Knowledge Graph . . . . . . . . . . . . . . . . . . . . . . 117
Zhiyuan Liu and Xianpei Han
6 Deep Learning in Machine Translation . . . . . . . . . . . . . . . . . . . . 147
Yang Liu and Jiajun Zhang
7 Deep Learning in Question Answering . . . . . . . . . . . . . . . . . . . . . 185
Kang Liu and Yansong Feng
8 Deep Learning in Sentiment Analysis . . . . . . . . . . . . . . . . . . . . . . 219
Duyu Tang and Meishan Zhang
9 Deep Learning in Social Computing . . . . . . . . . . . . . . . . . . . . . . . . 255
Xin Zhao and Chenliang Li
10 Deep Learning in Natural Language Generation from Images . . . . 289
Xiaodong He and Li Deng
11 Epilogue: Frontiers of NLP in the Deep Learning Era . . . . . . . . . . 309
Li Deng and Yang Liu
Glossary.. .... .... .... .... ..... .... .... .... .... .... ..... .... 327
xi