Table Of ContentStochastic Optimization
for Large-scale Machine
Learning
Stochastic Optimization
for Large-scale Machine
Learning
Vinod Kumar Chauhan, PhD
MATLAB® is a trademark of The MathWorks, Inc. and is used with permission. The MathWorks does not warrant the
accuracy of the text or exercises in this book. This book’s use or discussion of MATLAB® software or related products
does not constitute endorsement or sponsorship by The MathWorks of a particular pedagogical approach or particular
use of the MATLAB® software.
First edition published 2022
by CRC Press
6000 Broken Sound Parkway NW, Suite 300, Boca Raton, FL 33487-2742
and by CRC Press
2 Park Square, Milton Park, Abingdon, Oxon, OX14 4RN
© 2022 Vinod Kumar Chauhan
CRC Press is an imprint of Taylor & Francis Group, LLC
Reasonable efforts have been made to publish reliable data and information, but the author and publisher cannot as-
sume responsibility for the validity of all materials or the consequences of their use. The authors and publishers have
attempted to trace the copyright holders of all material reproduced in this publication and apologize to copyright holders
if permission to publish in this form has not been obtained. If any copyright material has not been acknowledged please
write and let us know so we may rectify in any future reprint.
Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced, transmitted, or
utilized in any form by any electronic, mechanical, or other means, now known or hereafter invented, including pho-
tocopying, microfilming, and recording, or in any information storage or retrieval system, without written permission
from the publishers.
For permission to photocopy or use material electronically from this work, access www.copyright.com or contact the
Copyright Clearance Center, Inc. (CCC), 222 Rosewood Drive, Danvers, MA 01923, 978-750-8400. For works that are
not available on CCC please contact [email protected]
Trademark notice: Product or corporate names may be trademarks or registered trademarks and are used only for iden-
tification and explanation without intent to infringe.
Library of Congress Cataloging-in-Publication Data
Names: Chauhan, Vinod Kumar, author.
Title: Stochastic optimization for large-scale machine learning / Vinod Kumar Chauhan.
Description: First edition. | Boca Raton : CRC Press, 2022. | Includes bibliographical references
and index.
Identifiers: LCCN 2021031740 | ISBN 9781032131757 (hardback) | ISBN 9781032146140 (pa-
perback) | ISBN 9781003240167 (ebook)
Subjects: LCSH: Machine learning--Statistical methods. | Big data. | Mathematical optimiza-
tion. | Stochastic processes.
Classification: LCC Q325.5 .C43 2022 | DDC 006.3/1--dc23 LC
record available at https://lccn.loc.gov/2021031740
ISBN: 978-1-032-13175-7 (hbk)
ISBN: 978-1-032-14614-0 (pbk)
ISBN: 978-1-003-24016-7 (ebk)
DOI: 10.1201/9781003240167
Publisher’s note: This book has been prepared from camera-ready copy provided by the authors.
Access the Support Material: www.routledge.com/9781032131757
To my parents...
Contents
List of Figures xiii
List of Tables xv
Preface xvii
Section I BACKGROUND
Chapter 1(cid:4) Introduction 3
1.1 LARGE-SCALEMACHINELEARNING 4
1.2 OPTIMIZATIONPROBLEMS 4
1.3 LINEARCLASSIFICATION 5
1.3.1 Support Vector Machine (SVM) 5
1.3.2 Logistic Regression 7
1.3.3 First and Second Order Methods 7
1.3.3.1 First Order Methods 7
1.3.3.2 Second Order Methods 8
1.4 STOCHASTICAPPROXIMATIONAPPROACH 8
1.5 COORDINATEDESCENTAPPROACH 8
1.6 DATASETS 8
1.7 ORGANIZATIONOFBOOK 9
Chapter 2(cid:4) Optimization Problem, Solvers, Challenges and Research
Directions 11
2.1 INTRODUCTION 11
2.1.1 Contributions 13
2.2 LITERATURE 13
2.3 PROBLEMFORMULATIONS 15
2.3.1 Hard Margin SVM (1992) 15
2.3.2 Soft Margin SVM (1995) 16
vii
viii (cid:4) Contents
2.3.3 One-versus-Rest (1998) 17
2.3.4 One-versus-One (1999) 18
2.3.5 Least Squares SVM (1999) 19
2.3.6 ν-SVM (2000) 19
2.3.7 Smooth SVM (2001) 20
2.3.8 Proximal SVM (2001) 21
2.3.9 Crammer Singer SVM (2002) 22
2.3.10 Eν-SVM (2003) 23
2.3.11 Twin SVM (2007) 23
2.3.12 Capped l -norm SVM (2017) 24
p
2.4 PROBLEMSOLVERS 29
2.4.1 Exact Line Search Method 31
2.4.2 Backtracking Line Search 31
2.4.3 Constant Step Size 32
2.4.4 Lipschitz and Strong Convexity Constants 32
2.4.5 Trust Region Method 32
2.4.6 Gradient Descent Method 32
2.4.7 Newton Method 33
2.4.8 Gauss-Newton Method 34
2.4.9 Levenberg-Marquardt Method 34
2.4.10 Quasi-Newton Method 35
2.4.11 Subgradient Method 35
2.4.12 Conjugate Gradient Method 36
2.4.13 Truncated Newton Method 36
2.4.14 Proximal Gradient Method 37
2.4.15 Recent Algorithms 37
2.5 COMPARATIVESTUDY 38
2.5.1 Results from Literature 38
2.5.2 Results from Experimental Study 39
2.5.2.1 Experimental Setup and Implementation Details 39
2.5.2.2 Results and Discussions 39
2.6 CURRENTCHALLENGESANDRESEARCHDIRECTIONS 41
2.6.1 Big Data Challenge 41
2.6.2 Areas of Improvement 41
2.6.2.1 Problem Formulations 41
2.6.2.2 Problem Solvers 42
Contents (cid:4) ix
2.6.2.3 Problem Solving Strategies/Approaches 43
2.6.2.4 Platforms/Frameworks 44
2.6.3 Research Directions 45
2.6.3.1 Stochastic Approximation Algorithms 45
2.6.3.2 Coordinate Descent Algorithms 45
2.6.3.3 Proximal Algorithms 45
2.6.3.4 Parallel/Distributed Algorithms 46
2.6.3.5 Hybrid Algorithms 46
2.7 CONCLUSION 46
Section II FIRST ORDER METHODS
Chapter 3(cid:4) Mini-batch and Block-coordinate Approach 51
3.1 INTRODUCTION 51
3.1.1 Motivation 52
3.1.2 Batch Block Optimization Framework (BBOF) 54
3.1.3 Brief Literature Review 56
3.1.4 Contributions 57
3.2 STOCHASTICAVERAGEADJUSTEDGRADIENT(SAAG)
METHODS 57
3.3 ANALYSIS 60
3.4 NUMERICALEXPERIMENTS 63
3.4.1 Experimental Setup 63
3.4.2 Convergence against Epochs 64
3.4.3 Convergence against Time 65
3.5 CONCLUSIONANDFUTURESCOPE 65
Chapter 4(cid:4) Variance Reduction Methods 67
4.1 INTRODUCTION 67
4.1.1 Optimization Problem 68
4.1.2 Solution Techniques for Optimization Problem 68
4.1.3 Contributions 69
4.2 NOTATIONSANDRELATEDWORK 70
4.2.1 Notations 70
4.2.2 Related Work 70
4.3 SAAG-I,IIANDPROXIMALEXTENSIONS 71
