Table Of ContentMastering Machine Learning
Algorithms
Expert techniques to implement popular machine learning
algorithms and fine-tune your models
Giuseppe Bonaccorso
BIRMINGHAM - MUMBAI
Mastering Machine Learning Algorithms
Copyright © 2018 Packt Publishing
All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form
or by any means, without the prior written permission of the publisher, except in the case of brief quotations
embedded in critical articles or reviews.
Every effort has been made in the preparation of this book to ensure the accuracy of the information presented.
However, the information contained in this book is sold without warranty, either express or implied. Neither the
author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to
have been caused directly or indirectly by this book.
Packt Publishing has endeavored to provide trademark information about all of the companies and products
mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy
of this information.
Commissioning Editor: Pravin Dhandre
Acquisition Editor: Divya Poojari
Content Development Editor: Eisha Dsouza
Technical Editors: Jovita Alva, Ishita Vora
Copy Editor: Safis Editing
Project Coordinator: Shweta H Birwatkar
Proofreader: Safis Editing
Indexer: Priyanka Dhadke
Graphics: Jisha Chirayil
Production Coordinator: Aparna Bhagat
First published: May 2018
Production reference: 1240518
Published by Packt Publishing Ltd.
Livery Place
35 Livery Street
Birmingham
B3 2PB, UK.
ISBN 978-1-78862-111-3
www.packtpub.com
To my parents, who always supported me in the journey of life!
– Giuseppe Bonaccorso
mapt.io
Mapt is an online digital library that gives you full access to over 5,000 books and videos, as
well as industry leading tools to help you plan your personal development and advance
your career. For more information, please visit our website.
Why subscribe?
Spend less time learning and more time coding with practical eBooks and Videos
from over 4,000 industry professionals
Improve your learning with Skill Plans built especially for you
Get a free eBook or video every month
Mapt is fully searchable
Copy and paste, print, and bookmark content
PacktPub.com
Did you know that Packt offers eBook versions of every book published, with PDF and
ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a
print book customer, you are entitled to a discount on the eBook copy. Get in touch with us
at [email protected] for more details.
At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a
range of free newsletters, and receive exclusive discounts and offers on Packt books and
eBooks.
Contributors
About the author
Giuseppe Bonaccorso is an experienced team leader/manager in AI, machine/deep learning
solution design, management, and delivery. He got his M.Sc.Eng. in Electronics in 2005
from University of Catania, Italy, and continued his studies at University of Rome Tor
Vergata and University of Essex, UK. His main interests include machine/deep learning,
reinforcement learning, big data, bio-inspired adaptive systems, cryptocurrencies, and
NLP.
I want to thank the people who have been close to me and have supported me, especially my
parents, who never stopped encouraging me.
About the reviewer
Francesco Azzola is an electronics engineer with over 15 years of experience in computer
programming. He is the author of Android Things Projects by Packt. He loves creating IoT
projects using Arduino, Raspberry Pi, Android, and other IoT platforms. He is interested in
convergence of IoT and mobile applications. He is certified in SCEA, SCWCD, and SCJP.
Packt is searching for authors like you
If you're interested in becoming an author for Packt, please visit authors.packtpub.com
and apply today. We have worked with thousands of developers and tech professionals,
just like you, to help them share their insight with the global tech community. You can
make a general application, apply for a specific hot topic that we are recruiting an author
for, or submit your own idea.
Table of Contents
Preface 1
Chapter 1: Machine Learning Model Fundamentals 8
Models and data 9
Zero-centering and whitening 11
Training and validation sets 13
Cross-validation 14
Features of a machine learning model 20
Capacity of a model 20
Vapnik-Chervonenkis capacity 22
Bias of an estimator 23
Underfitting 25
Variance of an estimator 27
Overfitting 27
The Cramér-Rao bound 28
Loss and cost functions 31
Examples of cost functions 36
Mean squared error 36
Huber cost function 36
Hinge cost function 37
Categorical cross-entropy 37
Regularization 38
Ridge 39
Lasso 41
ElasticNet 43
Early stopping 43
Summary 45
Chapter 2: Introduction to Semi-Supervised Learning 46
Semi-supervised scenario 46
Transductive learning 47
Inductive learning 48
Semi-supervised assumptions 48
Smoothness assumption 48
Cluster assumption 49
Manifold assumption 50
Generative Gaussian mixtures 51
Example of a generative Gaussian mixture 53
Weighted log-likelihood 59
Contrastive pessimistic likelihood estimation 60
Example of contrastive pessimistic likelihood estimation 63
Table of Contents
Semi-supervised Support Vector Machines (S3VM) 66
Example of S3VM 70
Transductive Support Vector Machines (TSVM) 76
Example of TSVM 77
Summary 84
Chapter 3: Graph-Based Semi-Supervised Learning 86
Label propagation 87
Example of label propagation 90
Label propagation in Scikit-Learn 94
Label spreading 96
Example of label spreading 98
Label propagation based on Markov random walks 100
Example of label propagation based on Markov random walks 101
Manifold learning 106
Isomap 106
Example of Isomap 109
Locally linear embedding 111
Example of locally linear embedding 113
Laplacian Spectral Embedding 115
Example of Laplacian Spectral Embedding 116
t-SNE 117
Example of t-distributed stochastic neighbor embedding 119
Summary 120
Chapter 4: Bayesian Networks and Hidden Markov Models 122
Conditional probabilities and Bayes' theorem 122
Bayesian networks 125
Sampling from a Bayesian network 126
Direct sampling 127
Example of direct sampling 128
A gentle introduction to Markov chains 130
Gibbs sampling 132
Metropolis-Hastings sampling 134
Example of Metropolis-Hastings sampling 135
Sampling example using PyMC3 137
Hidden Markov Models (HMMs) 142
Forward-backward algorithm 144
Forward phase 144
Backward phase 146
HMM parameter estimation 147
Example of HMM training with hmmlearn 149
Viterbi algorithm 151
Finding the most likely hidden state sequence with hmmlearn 153
Summary 154
Chapter 5: EM Algorithm and Applications 156
[ ii ]
