Natural Language Processing in Action: Understanding, analyzing, and generating text with Python

545 Pages·2019·9.7 MB·English

by  Hobson Lane

IN ACTION
Understanding, analyzing, and generating text with Python
Hobson Lane
Cole Howard
Hannes Max Hapke
Foreword by Dr. Arwen Griffioen
M A N N I N G
Chatbot Recirculating (Recurrent) Pipeline
Response
Text string
4. Execute
Generalize & classify
Scored
update models
responses
update objective
update dialog plan
select response
Structured
data
(feature Scored
1. Parse 2. Analyze 3. Generate
vector) statement
Tokenizers Check spelling Search
regular expressions check grammar templates
tag analyze sentiment FSM
NER analyze humanness MCMC
extract information analyze style RBM
Response
reduce dimensions CNN RNN
feature
GAN
vector
Database Scored
Statements statements
responses
scroes
user profiles
Possible
responses
Natural Language Processing in Action
ii
Natural Language
Processing in Action
Understanding, analyzing, and generating text with Python
HOBSON LANE
COLE HOWARD
HANNES MAX HAPKE
MANNING
SHELTER ISLAND
For online information and ordering of this and other Manning books, please visit
www.manning.com. The publisher offers discounts on this book when ordered in quantity.
For more information, please contact
Special Sales Department
Manning Publications Co.
20 Baldwin Road
PO Box 761
Shelter Island, NY 11964
Email: [email protected]
©2019 by Manning Publications Co. All rights reserved.
No part of this publication may be reproduced, stored in a retrieval system, or transmitted,
in any form or by means electronic, mechanical, photocopying, or otherwise, without prior
written permission of the publisher.
Many of the designations used by manufacturers and sellers to distinguish their products
are claimed as trademarks. Where those designations appear in the book, and Manning
Publications was aware of a trademark claim, the designations have been printed in initial
caps or all caps.
Recognizing the importance of preserving what has been written, it is Manning’s policy to have
the books we publish printed on acid-free paper, and we exert our best efforts to that end.
Recognizing also our responsibility to conserve the resources of our planet, Manning books
are printed on paper that is at least 15 percent recycled and processed without the use of
elemental chlorine.
Manning Publications Co. Acquisitions editor: Brian Sawyer
20 Baldwin Road Development editor: Karen Miller
PO Box 761 Technical development editor: René van den Berg
Shelter Island, NY 11964 Review editor: Ivan Martinovic´
Production editor: Anthony Calcara
Copy editor: Darren Meiss
Proofreader: Alyson Brener
Technical proofreader: Davide Cadamuro
Typesetter and cover designer: Marija Tudor
ISBN 9781617294631
Printed in the United States of America
1 2 3 4 5 6 7 8 9 10 – SP – 24 23 22 21 20 19
brief contents
PART 1 WORDY MACHINES .......................................................... 1
1 ■ Packets of thought (NLP overview) 3
2 ■ Build your vocabulary (word tokenization) 30
3 ■ Math with words (TF-IDF vectors) 70
4 ■ Finding meaning in word counts (semantic analysis) 97
PART 2 DEEPER LEARNING (NEURAL NETWORKS) ..................... 153
5 ■ Baby steps with neural networks
(perceptrons and backpropagation) 155
6 ■ Reasoning with word vectors (Word2vec) 181
7 ■ Getting words in order with convolutional
neural networks (CNNs) 218
8 ■ Loopy (recurrent) neural networks (RNNs) 247
9 ■ Improving retention with long short-term memory networks 274
10 ■ Sequence-to-sequence models and attention 311
PART 3 GETTING REAL (REAL-WORLD NLP CHALLENGES) ........ 337
11 ■ Information extraction (named entity extraction and
question answering) 339
12 ■ Getting chatty (dialog engines) 365
13 ■ Scaling up (optimization, parallelization, and
batch processing) 403
v
vi BRIEF CONTENTS
contents
foreword xiii
preface xv
acknowledgments xxi
about this book xxiv
about the authors xxvii
about the cover illustration xxix
PART 1 WORDY MACHINES ........................................... 1
1 Packets of thought (NLP overview) 3
1.1 Natural language vs. programming language 4
1.2 The magic 4
Machines that converse 5 ■ The math 6
1.3 Practical applications 8
1.4 Language through a computer’s “eyes” 9
The language of locks 10 ■ Regular expressions 11
A simple chatbot 12 ■ Another way 16
1.5 A brief overflight of hyperspace 19
1.6 Word order and grammar 21
1.7 A chatbot natural language pipeline 22
1.8 Processing in depth 25
1.9 Natural language IQ 27
vii
viii CONTENTS
2 Build your vocabulary (word tokenization) 30
2.1 Challenges (a preview of stemming) 32
2.2 Building your vocabulary with a tokenizer 33
Dot product 41 ■ Measuring bag-of-words overlap 42
A token improvement 43 ■ Extending your vocabulary with
n-grams 48 ■ Normalizing your vocabulary 54
2.3 Sentiment 62
VADER—A rule-based sentiment analyzer 64 ■ Naive Bayes 65
3 Math with words (TF-IDF vectors) 70
3.1 Bag of words 71
3.2 Vectorizing 76
Vector spaces 79
3.3 Zipf’s Law 83
3.4 Topic modeling 86
Return of Zipf 89 ■ Relevance ranking 90 ■ Tools 93
Alternatives 93 ■ Okapi BM25 95 ■ What’s next 95
4 Finding meaning in word counts (semantic analysis) 97
4.1 From word counts to topic scores 98
TF-IDF vectors and lemmatization 99 ■ Topic vectors 99
Thought experiment 101 ■ An algorithm for scoring topics 105
An LDA classifier 107
4.2 Latent semantic analysis 111
Your thought experiment made real 113
4.3 Singular value decomposition 116
U—left singular vectors 118 ■ S—singular values 119
VT—right singular vectors 120 ■ SVD matrix orientation 120
Truncating the topics 121
4.4 Principal component analysis 123
PCA on 3D vectors 125 ■ Stop horsing around and get back to
NLP 126 ■ Using PCA for SMS message semantic analysis 128
Using truncated SVD for SMS message semantic analysis 130
How well does LSA work for spam classification? 131
4.5 Latent Dirichlet allocation (LDiA) 134
The LDiA idea 135 ■ LDiA topic model for SMS messages 137
LDiA + LDA = spam classifier 140 ■ A fairer comparison:
32 LDiA topics 142