Jason Brownlee -Better deep learning

575 Pages·2019·9.42 MB·English

Better Deep Learning

Train Faster, Reduce Overfitting,
and Make Better Predictions

Jason Brownlee

i

Disclaimer

The information contained within this eBook is strictly for educational purposes. If you wish to apply
ideas contained in this eBook, you are taking full responsibility for your actions.
The author has made every effort to ensure the accuracy of the information within this book was
correct at time of publication. The author does not assume and hereby disclaims any liability to any
party for any loss, damage, or disruption caused by errors or omissions, whether such errors or
omissions result from accident, negligence, or any other cause.
No part of this eBook may be reproduced or transmitted in any form or by any means, electronic or
mechanical, recording or by any information storage and retrieval system, without written permission
from the author.

Acknowledgements

Special thanks to my proofreader Sarah Martin and my technical editors Andrei Cheremskoy, Michael
Sanderson, Arun Koshy.

Copyright

Better Deep Learning
© Copyright 2019 Jason Brownlee. All Rights Reserved.

Edition: v1.3

Contents

Copyright
i

Contents
ii

Preface
iii

Introduction
v

Welcome
v

Framework for Better Deep Learning
x

Diagnostic Learning Curves
xix

I
Better Learning
1

1
Improve Learning by Understanding Optimization
3

1.1
Neural Nets Learn a Mapping Function . . . . . . . . . . . . . . . . . . . . . . .
3

1.2
Learning Network Weights Is Hard
. . . . . . . . . . . . . . . . . . . . . . . . .
4

1.3
Key Features of the Error Surface . . . . . . . . . . . . . . . . . . . . . . . . . .
6

1.4
Navigating the Non-Convex Error Surface
. . . . . . . . . . . . . . . . . . . . .
8

1.5
Implications for Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
9

1.6
Components of the Learning Algorithm . . . . . . . . . . . . . . . . . . . . . . .
9

1.7
Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
11

1.8
Summary
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
12

2
Configure Capacity with Nodes and Layers
14

2.1
Neural Network Model Capacity . . . . . . . . . . . . . . . . . . . . . . . . . . .
14

2.2
Nodes and Layers Keras API . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
15

2.3
Model Capacity Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
17

2.4
Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
27

2.5
Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
27

2.6
Summary
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
28

ii

CONTENTS
iii

3
Configure Gradient Precision with Batch Size
29

3.1
Batch Size and Gradient Descent
. . . . . . . . . . . . . . . . . . . . . . . . . .
29

3.2
Gradient Descent Keras API . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
31

3.3
Batch Size Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
32

3.4
Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
47

3.5
Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
47

3.6
Summary
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
48

4
Configure What to Optimize with Loss Functions
49

4.1
Loss Functions
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
49

4.2
Regression Loss Functions Case Study
. . . . . . . . . . . . . . . . . . . . . . .
56

4.3
Binary Classification Loss Functions Case Study . . . . . . . . . . . . . . . . . .
64

4.4
Multiclass Classification Loss Functions Case Study . . . . . . . . . . . . . . . .
74

4.5
Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
84

4.6
Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
84

4.7
Summary
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
86

5
Configure Speed of Learning with Learning Rate
87

5.1
Learning Rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
87

5.2
Learning Rate Keras API
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
93

5.3
Learning Rate Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
95

5.4
Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118

5.5
Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118

5.6
Summary
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119

6
Stabilize Learning with Data Scaling
121

6.1
Data Scaling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121

6.2
Data Scaling scikit-learn API
. . . . . . . . . . . . . . . . . . . . . . . . . . . . 123

6.3
Data Scaling Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125

6.4
Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138

6.5
Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139

6.6
Summary
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139

7
Fix Vanishing Gradients with ReLU
141

7.1
Vanishing Gradients and ReLU
. . . . . . . . . . . . . . . . . . . . . . . . . . . 141

7.2
ReLU Keras API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152

7.3
ReLU Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152

7.4
Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164

7.5
Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165

7.6
Summary
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166

8
Fix Exploding Gradients with Gradient Clipping
167

8.1
Exploding Gradients and Clipping . . . . . . . . . . . . . . . . . . . . . . . . . . 167

8.2
Gradient Clipping Keras API
. . . . . . . . . . . . . . . . . . . . . . . . . . . . 169

8.3
Gradient Clipping Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170

8.4
Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178

8.5
Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179

CONTENTS
iv

8.6
Summary
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179

9
Accelerate Learning with Batch Normalization
180

9.1
Batch Normalization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180

9.2
Batch Normalization Keras API . . . . . . . . . . . . . . . . . . . . . . . . . . . 186

9.3
Batch Normalization Case Study
. . . . . . . . . . . . . . . . . . . . . . . . . . 189

9.4
Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198

9.5
Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198

9.6
Summary
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200

10 Deeper Models with Greedy Layer-Wise Pretraining
201

10.1 Greedy Layer-Wise Pretraining
. . . . . . . . . . . . . . . . . . . . . . . . . . . 201

10.2 Greedy Layer-Wise Pretraining Case Study . . . . . . . . . . . . . . . . . . . . . 204
10.3 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219
10.4 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219
10.5 Summary
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220

11 Jump-Start Training with Transfer Learning
221

11.1 Transfer Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221
11.2 Transfer Learning Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222
11.3 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240
11.4 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240
11.5 Summary
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241

II
Better Generalization
243

12 Fix Overfitting with Regularization
245

12.1 Problem of Model Generalization and Overfitting
. . . . . . . . . . . . . . . . . 245

12.2 Reduce Overfitting by Constraining Complexity . . . . . . . . . . . . . . . . . . 247
12.3 Regularization Methods for Neural Networks . . . . . . . . . . . . . . . . . . . . 248
12.4 Regularization Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . 249
12.5 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 250
12.6 Summary
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251

13 Penalize Large Weights with Weight Regularization
252

13.1 Weight Regularization
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252

13.2 Weight Regularization Keras API . . . . . . . . . . . . . . . . . . . . . . . . . . 258
13.3 Weight Regularization Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . 260
13.4 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 268
13.5 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269
13.6 Summary
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270

14 Sparse Representations with Activity Regularization
272

14.1 Activity Regularization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272
14.2 Activity Regularization Keras API
. . . . . . . . . . . . . . . . . . . . . . . . . 276

14.3 Activity Regularization Case Study . . . . . . . . . . . . . . . . . . . . . . . . . 278

CONTENTS
v

14.4 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287
14.5 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 288
14.6 Summary
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289

15 Force Small Weights with Weight Constraints
290

15.1 Weight Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 290
15.2 Weight Constraints Keras API . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293
15.3 Weight Constraints Case Study
. . . . . . . . . . . . . . . . . . . . . . . . . . . 295

15.4 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302
15.5 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303
15.6 Summary
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304

16 Decouple Layers with Dropout
305

16.1 Dropout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305
16.2 Dropout Keras API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 310
16.3 Dropout Case Study
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 314

16.4 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 320
16.5 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 321
16.6 Summary
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 322

17 Promote Robustness with Noise
323

17.1 Noise Regularization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323
17.2 Noise Regularization Keras API . . . . . . . . . . . . . . . . . . . . . . . . . . . 328
17.3 Noise Regularization Case Study
. . . . . . . . . . . . . . . . . . . . . . . . . . 330

17.4 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 341
17.5 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 341
17.6 Summary
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343

18 Halt Training at the Right Time with Early Stopping
344

18.1 Early Stopping
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 344

18.2 Early Stopping Keras API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 350
18.3 Early Stopping Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353
18.4 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 364
18.5 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 364
18.6 Summary
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 366

III
Better Predictions
367

19 Reduce Model Variance with Ensemble Learning
369

19.1 High Variance of Neural Network Models . . . . . . . . . . . . . . . . . . . . . . 369
19.2 Reduce Variance Using an Ensemble of Models . . . . . . . . . . . . . . . . . . . 370
19.3 How to Ensemble Neural Network Models
. . . . . . . . . . . . . . . . . . . . . 371

19.4 Summary of Ensemble Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . 375
19.5 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 376
19.6 Summary
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 377

CONTENTS
vi

20 Combine Models From Multiple Runs with Model Averaging Ensemble
379

20.1 Model Averaging Ensemble . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 379
20.2 Ensembles in Keras . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 380
20.3 Model Averaging Ensemble Case Study . . . . . . . . . . . . . . . . . . . . . . . 382
20.4 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 396
20.5 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 396
20.6 Summary
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 397

21 Contribute Proportional to Trust with Weighted Average Ensemble
398

21.1 Weighted Average Ensemble . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 398
21.2 Weighted Average Ensemble Case Study . . . . . . . . . . . . . . . . . . . . . . 399
21.3 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 417
21.4 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 417
21.5 Summary
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 419

22 Fit Models on Different Samples with Resampling Ensembles
420

22.1 Resampling Ensembles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 420
22.2 Resampling Ensembles Case Study
. . . . . . . . . . . . . . . . . . . . . . . . . 422

22.3 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 440
22.4 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 441
22.5 Summary
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 442

23 Models from Contiguous Epochs with Horizontal Voting Ensembles
443

23.1 Horizontal Voting Ensemble . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 443
23.2 Horizontal Voting Ensembles Case Study . . . . . . . . . . . . . . . . . . . . . . 444
23.3 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 456
23.4 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 456
23.5 Summary
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 457

24 Cyclic Learning Rate and Snapshot Ensembles
458

24.1 Snapshot Ensembles
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 458

24.2 Snapshot Ensembles Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . 460
24.3 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 479
24.4 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 480
24.5 Summary
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 480

25 Learn to Combine Predictions with Stacked Generalization Ensemble
482

25.1 Stacked Generalization Ensemble
. . . . . . . . . . . . . . . . . . . . . . . . . . 482

25.2 Stacked Generalization Ensemble Case Study . . . . . . . . . . . . . . . . . . . . 484
25.3 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 500
25.4 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 500
25.5 Summary
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 502

26 Combine Model Parameters with Average Model Weights Ensemble
503

26.1 Average Model Weight Ensemble
. . . . . . . . . . . . . . . . . . . . . . . . . . 503

26.2 Average Model Weight Ensemble Case Study . . . . . . . . . . . . . . . . . . . . 505
26.3 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 523

CONTENTS
vii

26.4 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 523
26.5 Summary
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 524

IV
Appendix
525

A Getting Help
526

A.1 Applied Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 526
A.2 Official Keras Destinations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 526
A.3 Where to Get Help with Keras . . . . . . . . . . . . . . . . . . . . . . . . . . . . 527
A.4 How to Ask Questions
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 527

A.5 Contact the Author . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 528

B How to Setup Your Workstation
529

B.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 529
B.2 Download Anaconda
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 529

B.3 Install Anaconda
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 531

B.4 Start and Update Anaconda . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 533
B.5 Install Deep Learning Libraries
. . . . . . . . . . . . . . . . . . . . . . . . . . . 536

B.6 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 537
B.7 Summary
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 537

V
Conclusions
538

How Far You Have Come
539

Preface

Modern open source libraries for developing neural network models are amazing. Gone are the
days where we might spend weeks debugging the translation of poorly documented mathematics
into code in the hopes of getting even the simplest model running. Today, using libraries like
Keras, we can define and begin fitting a Multilayer Perceptron, Convolutional or even Recurrent
Neural Network model of arbitrary complexity in minutes.
While defining and fitting models has become trivial, getting good performance with a
neural network model on a specific problem remains challenging. Traditionally, configuring
neural networks in order to get good performance was referred to as a dark art. This is because
there are no clear rules on how to best prepare data and configure a model for a given problem.
Instead, experience must be developed over time from working on many different projects.
Nevertheless, neural networks have been used in academia and industry for decades now and
there are a suite of standard techniques, tips, and configurations that you can use to greatly
increase the likelihood of getting better-than-average performance with a neural network model.
I wrote this book to pull together the best classical and modern techniques in order to
provide a playbook that you can use to get better performance on your next project using deep
learning neural networks. A lot has changed in the last 5 to 10 years and there are activation
functions, regularization methods, and even new ensemble methods that result in remarkably
faster learning, lower generalization error, and more robust results. I hope that you’re as excited
as me about the journey ahead.

Jason Brownlee
2019

viii

Introduction

ix

Welcome

Welcome to Better Deep Learning. Deep learning neural networks have become easy to define
and fit, but it remains challenging to achieve good predictive modeling performance. Neural
networks have been studied in academia and used in industry for decades, and there is a wealth
of techniques, tips, and model configurations that are known to result in better than average
performance. In addition, the last 5 to 10 years has seen the development and adoption of
modern network configurations, regularization techniques, and ensemble algorithms that result
in superior performance. I designed this book to tie together classical and modern techniques
into a single playbook and teach you step-by-step how to improve the performance of deep
learning neural network models on your predictive modeling problems.

Who Is This Book For?

Before we get started, let’s make sure you are in the right place. This book is for developers that
know some applied machine learning and some deep learning. Maybe you want or need to start
using deep learning on your research project or on a project at work. This guide was written to
help you do that quickly and efficiently by compressing years of knowledge and experience into
a laser-focused course of hands-on tutorials. The lessons in this book assume a few things about
you, such as:

ˆ You know your way around basic Python for programming.

ˆ You know your way around basic NumPy for array manipulation.

ˆ You know your way around basic Keras for deep learning.

This guide was written in the top-down and results-first machine learning style that you’re
used to from MachineLearningMastery.com.

About Your Outcomes

This book will teach you how to get results as a machine learning practitioner interested in getting
the most out of deep learning models on your own predictive modeling projects. Techniques
are demonstrated in the context of small well-understood predictive modeling problems and
Multilayer Perceptron neural network models, but can easily be applied to a wide range of
problems and with a suite of different types of neural network models. After reading and working
through this book, you will know:

x

xi

ˆ A checklist of techniques that you can use to improve the performance of deep learning
neural network models on your own predictive modeling problems.

ˆ How to accelerate learning through better configured stochastic gradient descent batch
size, loss functions, learning rates, and to avoid exploding gradients via gradient clipping.

ˆ How to accelerate learning through correct data scaling, batch normalization, and use of
modern activation functions such as the rectified linear activation function.

ˆ How to accelerate learning through choosing better initial weights with greedy layer-wise
pretraining and transfer learning.

ˆ A gentle introduction to the problem of overfitting and a tour of regularization techniques.

ˆ How to reduce overfitting by updating the loss function using techniques such as weight
regularization, weight constraints, and activation regularization.

ˆ How to reduce overfitting using techniques such as dropout, the addition of noise, and
early stopping.

ˆ A gentle introduction to how to combine the predictions from multiple models and a tour
of ensemble learning techniques.

ˆ How to combine the predictions from multiple different models using techniques such
as weighted averaging ensembles and stacked generalization ensembles, also known as
blending.

ˆ How to combine the predictions from multiple models saved during a single training run
with techniques such as horizontal ensembles and snapshot ensembles.

This new understanding of applied deep learning methods will impact your practice of
working through predictive modeling problems in the following ways:

ˆ Confidently diagnose poor model training and problems such as premature convergence
and accelerate the model training process using one or a combination of modifications to
the learning algorithm.

ˆ Confidently diagnose cases of overfitting the training dataset and reduce generalization
error using one or a combination of modifications of the model, loss function, or learning
algorithm.

ˆ Confidently diagnose high variance in a final model and improve the average predictive
skill by combining the predictions from multiple models trained over a single or multiple
training runs.

This book will teach you how to get good results, quickly, but will NOT teach you how to
be a research scientist nor will it teach you all the theory behind why specific methods work.
For that, I would recommend good research papers and textbooks. See the Further Reading
section at the end of each tutorial for a good starting point.

xii

How to Read This Book

This book was written to be read linearly, from start to finish. That being said, if you know the
basics and need help with a specific problem type or technique, then you can flip straight to
that section and get started. This book was designed for you to read on your workstation, on
the screen. Not away from the computer or on a tablet or eReader. My hope is that you have
the book open right next to your editor and run the examples as you read about them.
This book is not intended to be read passively or be placed in a folder as a reference text.
It is a playbook, a workbook, and a guidebook intended for you to learn by doing and then
apply your new understanding to your own predictive modeling projects. To get the most out of
the book, I would recommend playing with the examples in each tutorial. Extend them, break
them, then fix them. Try some of the extensions presented at the end of each lesson and let me
know how you do.

About the Book Structure

This book was designed around three main activities for getting better results with deep learning
models: better or faster learning, better generalization to new data, and better predictions when
using final models. There are a lot of things you could learn about getting better results from
neural network models, from theory to applications to APIs. My goal is to take you straight
to getting results with laser-focused tutorials. I designed the tutorials to focus on how to get
things done. They give you the tools to both rapidly understand and apply each technique to
your own predictive modeling problems.
Each of the tutorials are designed to take you about one hour to read through and complete,
excluding the extensions and further reading. You can choose to work through the lessons one
per day, one per week, or at your own pace. I think momentum is critically important, and this
book is intended to be read and used, not to sit idle. I would recommend picking a schedule
and sticking to it. The tutorials are divided into three parts:

ˆ Part 1: Better Learning. Discover the techniques to improve and accelerate the process
used to learn or optimize the weights of a neural network model.

ˆ Part 2: Better Generalization. Discover the techniques to reduce overfitting of the
training dataset and improve the generalization of models on new data.

ˆ Part 3: Better Predictions. Discover the techniques to improve the performance of
final models when used to make predictions on new data.

Each part targets a specific approach to improving model performance, and each tutorial
targets a specific learning outcome for a technique. This acts as a filter to ensure you are only
focused on the things you need to know to get to a specific result and do not get bogged down in
the math or near-infinite number of configuration parameters. The tutorials were not designed
to teach you everything there is to know about each of the techniques. They were designed to
give you an understanding of how they work and how to use them on your projects the fastest
way I know how: to learn by doing.

xiii

About Python Code Examples

The code examples were carefully designed to demonstrate the purpose of a given lesson. For
this reason, the examples are highly targeted.

ˆ Models were demonstrated on small contrived datasets to give you the context and
confidence to bring the techniques to your own projects.

ˆ Model configurations used were discovered through trial and error and are skillful but
not optimized. This leaves the door open for you to explore new and possibly better
configurations.

ˆ Code examples are complete and standalone. The code for each lesson will run as-is with
no code from prior lessons or third parties required beyond the installation of the required
packages.

A complete working example is presented with each tutorial for you to inspect and copy
and paste. All source code is also provided with the book and I would recommend running
the provided files whenever possible to avoid any copy-paste issues. The provided code was
developed in a text editor and intended to be run on the command line. No special IDE or
notebooks are required. If you are using a more advanced development environment and are
having trouble, try running the example from the command line instead.
Neural network algorithms are stochastic. This means that they will make different predictions
when the same model configuration is run on the same training data. On top of that, each
experimental problem in this book is based around generating stochastic predictions. As a
result, this means you will not get exactly the same sample output presented in this book. This
is by design. I want you to get used to the stochastic nature of the neural network algorithms.
If this bothers you, please note:

ˆ You can re-run a given example a few times and your results should be close to the values
reported.

ˆ You can make the output consistent by fixing the random number seed.

ˆ You can develop a robust estimate of the skill of a model by fitting and evaluating it
multiple times and taking the average of the final skill score (highly recommended).

All code examples were tested on a POSIX-compatible machine with Python 3 and Keras 2.
All code examples will run on modest and modern computer hardware and were executed on a
CPU. No GPUs are required to run the presented examples, although a GPU would make the
code run faster. I am only human and there may be a bug in the sample code. If you discover a
bug, please let me know so I can fix it and update the book and send out a free update.

About Further Reading

Each lesson includes a list of further reading resources. This may include:

ˆ Research papers.

xiv

ˆ Books and book chapters.

ˆ Webpages.

ˆ API documentation.

Wherever possible, I try to list and link to the relevant API documentation for key objects
and functions used in each lesson so you can learn more about them. When it comes to research
papers, I try to list papers that are first to use a specific technique or first in a specific problem
domain. These are not required reading, but can give you more technical details, theory, and
configuration details if you’re looking for it. Wherever possible, I have tried to link to the freely
available version of the paper on arxiv.org. You can search for and download any of the papers
listed on Google Scholar Search, scholar.google.com. Wherever possible, I have tried to link to
books on Amazon.
I don’t know everything, and if you discover a good resource related to a given lesson, please
let me know so I can update the book.

About Getting Help

You might need help along the way. Don’t worry; you are not alone.

ˆ Help with a Technique? If you need help with the technical aspects of a specific model
or method, see the Further Reading sections at the end of each lesson.

ˆ Help with Keras? If you need help with using the Keras library, see the list of resources
in Appendix A.

ˆ Help with your workstation? If you need help setting up your environment, I would
recommend using Anaconda and following my tutorial in Appendix B.

ˆ Help in general? You can shoot me an email. My details are in Appendix A.

Next

Are you ready? Let’s dive in! In the next tutorial, you will discover a framework that you can
use to diagnose problems with your deep learning neural network and techniques that you can
use to address each identified problem.