Scaling Python with Ray: Adventures in Cloud and Serverless Patterns

269 Pages·2023·3.481 MB·English

by  Holden Karau, Boris Lublinsky

Scaling Python
with Ray
Adventures in Cloud and Serverless Patterns
L
uK
ba
linra
su
ky &

Holden Karau &
Boris Lublinsky
Foreword by Robert Nishihara
Scaling Python with Ray
Serverless computing enables developers to concentrate
solely on their applications rather than worry about where “Scaling Python with
they’ve been deployed. With the Ray general-purpose Ray is a concise and
serverless implementation in Python, programmers and data
practical guide to
scientists can hide servers, implement stateful applications,
adopting Ray and using
support direct communication between tasks, and access
it effectively. Informed
hardware accelerators.
by years of industry
In this book, experienced software architecture practitioners
experience in data
Holden Karau and Boris Lublinsky show you how to scale
existing Python applications and pipelines, allowing you to systems and distributed
stay in the Python ecosystem while reducing single points computing, Holden
of failure and manual scheduling. Scaling Python with Ray is and Boris deliver the
ideal for software architects and developers eager to explore
indispensable guide that
successful case studies and learn more about decision and
users of Ray need.”
measurement effectiveness.
—Dean Wampler, PhD
If your data processing or server application has grown Engineering Director, Accelerated
beyond what a single computer can handle, this book is for Discovery Platform, IBM Research
you. You’ll explore distributed processing (the pure Python
implementation of serverless) and learn how to:
Holden Karau is a queer transgender
• Implement stateful applications with Ray actors Canadian, Apache Spark committer,
Apache Software Foundation member,
• Build workflow management in Ray
and an active open source contributor.
• Use Ray as a unified system for batch and stream processing Boris Lublinsky is a chief architect for
• Apply advanced data processing with Ray IBM’s Discovery Accelerator Platform,
where he specializes in Kubernetes,
• Build microservices with Ray serverless, workflows, and complex
• Implement reliable Ray applications systems design.
DATA Twitter: @oreillymedia
linkedin.com/company/oreilly-media
US $59.99 CAN $74.99 youtube.com/oreillymedia
ISBN: 978-1-098-11880-8
Scaling Python with Ray
Adventures in Cloud and Serverless Patterns
Holden Karau and Boris Lublinsky
Foreword by Robert Nishihara
BBeeiijjiinngg BBoossttoonn FFaarrnnhhaamm SSeebbaassttooppooll TTookkyyoo
Scaling Python with Ray
by Holden Karau and Boris Lublinsky
Copyright © 2023 Holden Karau and Boris Lublinsky. All rights reserved.
Printed in the United States of America.
Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472.
O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are
also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional
sales department: 800-998-9938 or [email protected].
Acquisitions Editors: Aaron Black Proofreader: Justin Billing
and Jessica Haberman Indexer: nSight, Inc.
Development Editor: Virginia Wilson Interior Designer: David Futato
Production Editor: Gregory Hyman Cover Designer: Karen Montgomery
Copyeditor: Sharon Wilkey Illustrator: Kate Dullea
December 2022: First Edition
Revision History for the First Edition
2022-11-29: First Release
See http://oreilly.com/catalog/errata.csp?isbn=9781098118808 for release details.
The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Scaling Python with Ray, the cover
image, and related trade dress are trademarks of O’Reilly Media, Inc.
The views expressed in this work are those of the authors, and do not represent the publisher’s views.
While the publisher and the authors have used good faith efforts to ensure that the information and
instructions contained in this work are accurate, the publisher and the authors disclaim all responsibility
for errors or omissions, including without limitation responsibility for damages resulting from the use
of or reliance on this work. Use of the information and instructions contained in this work is at your
own risk. If any code samples or other technology this work contains or describes is subject to open
source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use
thereof complies with such licenses and/or rights.
978-1-098-11880-8
[LSI]
Table of Contents
Foreword. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi
1. What Is Ray, and Where Does It Fit?. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Why Do You Need Ray? 2
Where Can You Run Ray? 3
Running Your Code with Ray 5
Where Does It Fit in the Ecosystem? 5
Big Data / Scalable DataFrames 7
Machine Learning 8
Workflow Scheduling 8
Streaming 9
Interactive 9
What Ray Is Not 9
Conclusion 10
2. Getting Started with Ray (Locally). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
Installation 11
Installing for x86 and M1 ARM 12
Installing (from Source) for ARM 12
Hello Worlds 13
Ray Remote (Task/Futures) Hello World 13
Data Hello World 16
Actor Hello World 17
Conclusion 19
iii
3. Remote Functions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
Essentials of Ray Remote Functions 22
Composition of Remote Ray Functions 27
Ray Remote Best Practices 30
Bringing It Together with an Example 31
Conclusion 33
4. Remote Actors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
Understanding the Actor Model 35
Creating a Basic Ray Remote Actor 36
Implementing the Actor’s Persistence 41
Scaling Ray Remote Actors 45
Ray Remote Actors Best Practices 50
Conclusion 51
5. Ray Design Details. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
Fault Tolerance 53
Ray Objects 56
Serialization/Pickling 59
cloudpickle 59
Apache Arrow 61
Resources / Vertical Scaling 62
Autoscaler 64
Placement Groups: Organizing Your Tasks and Actors 65
Namespaces 70
Managing Dependencies with Runtime Environments 70
Deploying Ray Applications with the Ray Job API 71
Conclusion 74
6. Implementing Streaming Applications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
Apache Kafka 76
Basic Kafka Concepts 77
Kafka APIs 80
Using Kafka with Ray 81
Scaling Our Implementation 87
Building Stream-Processing Applications with Ray 88
Key-Based Approach 89
Key-Independent Approach 95
Going Beyond Kafka 95
Conclusion 96
iv | Table of Contents
7. Implementing Microservices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
Understanding Microservice Architecture in Ray 97
Deployment 98
Additional Deployment Capabilities 101
Deployment Composition 104
Using Ray Serve for Model Serving 105
Simple Model Service Example 106
Considerations for Model-Serving Implementations 107
Speculative Model Serving Using the Ray Microservice Framework 109
Conclusion 111
8. Ray Workflows. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
What Is Ray Workflows? 113
How Is It Different from Other Solutions? 114
Ray Workflows Features 114
What Are the Main Features? 114
Workflow Primitives 115
Working with Basic Workflow Concepts 116
Workflows, Steps, and Objects 116
Dynamic Workflows 117
Virtual Actors 118
Workflows in Real Life 118
Building Workflows 118
Managing Workflows 119
Building a Dynamic Workflow 121
Building Workflows with Conditional Steps 121
Handling Exceptions 122
Handling Durability Guarantees 123
Extending Dynamic Workflows with Virtual Actors 124
Integrating Workflows with Other Ray Primitives 129
Triggering Workflows (Connecting to Events) 130
Working with Workflow Metadata 131
Conclusion 133
9. Advanced Data with Ray. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
Creating and Saving Ray Datasets 136
Using Ray Datasets with Different Tools 138
Using Tools on Ray Datasets 139
pandas-like DataFrames with Dask 140
Indexing 141
Table of Contents | v
Shuffles 142
Embarrassingly Parallel Operations 148
Working with Multiple DataFrames 149
What Does Not Work 151
What’s Slower 152
Handling Recursive Algorithms 152
What Other Functions Are Different 153
pandas-like DataFrames with Modin 153
Big Data with Spark 154
Working with Local Tools 154
Using Built-in Ray Dataset Operations 155
Implementing Ray Datasets 158
Conclusion 159
10. How Ray Powers Machine Learning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
Using scikit-learn with Ray 161
Using Boosting Algorithms with Ray 165
Using XGBoost 166
Using LightGBM 167
Using PyTorch with Ray 169
Reinforcement Learning with Ray 174
Hyperparameter Tuning with Ray 180
Conclusion 186
11. Using GPUs and Accelerators with Ray. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
What Are GPUs Good At? 187
The Building Blocks 188
Higher-Level Libraries 188
Acquiring and Releasing GPU and Accelerator Resources 189
Ray’s ML Libraries 190
Autoscaler with GPUs and Accelerators 190
CPU Fallback as a Design Pattern 191
Other (Non-GPU) Accelerators 192
Conclusion 192
12. Ray in the Enterprise. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
Ray Dependency Security Issues 193
Interacting with the Existing Tools 193
Using Ray with CI/CD Tools 194
Authentication with Ray 194
Multitenancy on Ray 195
vi | Table of Contents
Credentials for Data Sources 196
Permanent Versus Ephemeral Clusters 196
Ephemeral Clusters 197
Permanent Clusters 197
Monitoring 198
Instrumenting Your Code with Ray Metrics 201
Wrapping Custom Programs with Ray 203
Conclusion 204
A. Space Beaver Case Study: Actors, Kubernetes, and More. . . . . . . . . . . . . . . . . . . . . . . . . 205
B. Installing and Deploying Ray. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221
C. Debugging with Ray. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233
Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241
Table of Contents | vii