The Apache Ignite book

111 Pages·2017·25.32 MB·English

The Apache Ignite book
The next phase of the distributed systems
Shamim Bhuiyan and Michael Zheludkov
Thisbookisforsaleathttp://leanpub.com/ignitebook
Thisversionwaspublishedon2023-02-12
ISBN978-0-359-43937-9
ThisisaLeanpubbook.LeanpubempowersauthorsandpublisherswiththeLean
Publishingprocess.LeanPublishingistheactofpublishinganin-progressebookusing
lightweighttoolsandmanyiterationstogetreaderfeedback,pivotuntilyouhavetheright
bookandbuildtractiononceyoudo.
©2018-2023ShamimBhuiyan
Tweet This Book!
PleasehelpShamimBhuiyanandMichaelZheludkovbyspreadingthewordaboutthis
bookonTwitter!
Thesuggestedtweetforthisbookis:
Justpurchased”TheApacheIgniteBook”https://leanpub.com/ignitebookby@shamim_ru
#ApacheIgnite#IMDG#NoSQL#BigData#caching
TomyMother&Brothers,thankyouforyourunconditionallove.
Contents
Abouttheauthors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Chapter4.Architecturedeepdive . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Understandingtheclustertopology:shared-nothingarchitecture . . . . . . . . . . 1
Clientandservernode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Embeddedwiththeapplication . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Clientandtheservernodesinthesamehost . . . . . . . . . . . . . . . . . . . 6
RunningmultiplenodeswithinsingleJVM . . . . . . . . . . . . . . . . . . . . 6
Realclustertopology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
DatapartitioninginIgnite. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Understandingdatadistribution:DHT . . . . . . . . . . . . . . . . . . . . . . . 9
Rendezvoushashing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
Durablememoryarchitecture. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
Page . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
DataPage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
IndexpagesandB+trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
Segments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
Region. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
Igniteread/writepath . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
Write-Ahead-Log(WAL) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
Baselinetopology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
Automaticclusteractivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
Split-brainprotection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
Fastrebalancingandit’spitfalls . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
Chapter5.Intelligentcaching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
Smartcaching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
Cachingbestpractices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
Designpatterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
CONTENTS
Basicterms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
Databasecaching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
Chapter8.Streamingandcomplexeventprocessing . . . . . . . . . . . . . . . . . . 50
KafkaStreamer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
IgniteSourceConnector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
Chapter10.Managementandmonitoring . . . . . . . . . . . . . . . . . . . . . . . . . 56
ManagingIgnitecluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
MonitoringIgnitecluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
VisualVM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
Grafana . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
About the authors
ShamimBhuiyaniscurrentlyworkingasanEnterprisearchitect;wherehe’sresponsiblefor
designingandbuildingouthighlyscalable,andhigh-loadmiddlewaresolutions.Hereceived
hisPh.D.inComputerSciencefromtheUniversityofVladimir,Russiain2007.Hehasbeen
in the IT field for over 18 years and is specialized in Middleware solutions, Big Data and
Data science. Also, he is a former SOA solution designer, speaker, and Big data evangelist.
Actively participates in the development and designing of high-performance software for
IT,telecommunicationandthebankingindustry.Insparetimes,heusuallywritestheblog
frommyworkshop¹andsharesideaswithothers.
MichaelZheludkovisaseniorprogrammeratATConsulting.HegraduatedfromtheBau-
man Moscow State Technical University in 2002. Lecturer at BMSTU since 2013, delivering
courseParallelprogramminganddistributedsystems.
¹http://frommyworkshop.blogspot.ru/
Chapter 4. Architecture deep dive
ApacheIgniteisanopen-sourcememory-centricdistributeddatabase,cachingandcomput-
ingplatform.Itwasdesignedasanin-memorydatagridfordevelopingahigh-performance
softwaresystemfromthebeginning.Soitscorearchitecturedesignisslightlydifferentfrom
thatofthetraditionalNoSQLdatabases,abletosimplifythebuildingofmodernapplications
withaflexibledatamodelandsimplerhighavailabilityandhighscalability.
To understand how to properly design an application with any databases or framework,
youmustfirstunderstandthearchitectureofthedatabaseorframeworkitself.Bygettinga
better idea of the system, you can solve different problems in your enterprise architecture
landscape, can select a comprehensive database or framework that is appropriate for your
applicationandcangetmaximumbenefitsfromthesystem.Thischaptergivesyoualookat
theApacheIgnitearchitectureandcorecomponentstohelpyoufigureoutthekeyreasons
behindIgnite’ssuccessoverotherplatforms.
Understanding the cluster topology:
shared-nothing architecture
Apache Ignite is a grid technology, and its design implies that the entire system is both
inherently available and massively scalable. Grid computing is a technology in which we
utilize the resources of many computers (commodity, on-premise, VM, etc.) in a network
towardssolvingasinglecomputingprobleminparallelfashion.
Note that there is often some confusion about the difference between grid and cluster.
Grid computing is very similar to cluster computing, the big difference being that cluster
computing consists of homogeneous resources, while grids are heterogeneous. Computers
that are part of a grid can run different operating systems and have different hardware,
whereasclustercomputersallhavethesamehardwareandOS.Agridcanmakeuseofspare
computing power on a desktop computer, while the machines in a cluster are dedicated to
workingasasingleunitandnothingelse.Throughoutthisbook,weusethetermsgrid and
cluster interchangeably.
Apache Ignite also provides a shared-nothing architecture² where multiple identical nodes
²https://en.wikipedia.org/wiki/Shared-nothing_architecture
Chapter4.Architecturedeepdive 2
formaclusterwithnosinglemasterorcoordinator.Allnodesinashared-nothingclusterare
identicalandruntheexactsameprocess.IntheIgnitegrid,nodescanbeaddedorremoved
nondisruptively to increase (or decrease) the amount of RAM available. Ignite internode
communication allows all nodes to receive updates quickly without having any master
coordinator. Nodes communicate using peer-to-peer message passing. The Apache Ignite
grid is sufficiently resilient, allowing the nondisruptive automated detection and recovery
ofasinglenodeormultiplenodes.
Onthemostfundamentallevel,allnodesintheIgniteclusterfallintooneoftwocategories:
client andserver.Thereisabigdifferencebetweenthetwotypesofnodes,andtheycanbe
deployedindifferentways.Intherestofthissection,wewilltalkaboutthetopologyofthe
Ignitegridandhowitcanbedeployedinreallife.
Client and server node
An Ignite node is a single Ignite process running in a JVM. Apache Ignite nodes have an
optional notion of client and server nodes as we mentioned before. Often, an Ignite client
nodealsoaddressesasanativeclientnode.BothclientandservernodesarepartofIgnite’s
physicalgridandareinterconnectedwitheachother.Theclientandservernodeshavethe
followingcharacteristics.
Node Description
Server 1.Actsasacontainerforstoringdataandcomputing.Aservernodecontainsdata,
participatesincaching,computingandstreaming.2.Generallystartsasastandalone
Javaprocess.
Client 1.Actsasanentrypointtorunoperationslikeput/getintothecache.2.Canstore
portionsofdatainthenearcache,whichisasmallerlocalcachethatstoresmost
recentlyandmostfrequentlyaccesseddata.3.Itisalsousedtodeploycomputeand
servicetaskstotheservernodesandcanparticipateincomputationtasks(optional).4.
Usuallyembeddedwiththeapplicationcode.
Tip
Youoftenencounterthetermdatanode intheIgnitedocumentation.Theterms
datanodeandservernoderefertothesamethingandareusedinterchangeably.
All nodes in the Ignite grid start as server nodes by default, and client nodes need to be
explicitly enabled.YoucanimaginetheIgniteclientnodeasathickclient (alsocalledafat
client,e.g.,OracleOCI8).WheneveraclientnodeconnectstotheIgnitegridorcluster,itis
Chapter4.Architecturedeepdive 3
awareofthegridtopology(datapartitionsforeachnode)andisabletosendarequesttothe
particular node to retrieve data. You can configure an Ignite node to be either a client or a
serverviaaSpringorJavaconfiguration,asshownbelow.
Springconfiguration:
Listing4.1
<beanclass="org.apache.ignite.configuration.IgniteConfiguration">
...
<!--Enableclientmode.-->
<propertyname="clientMode"value="true"/>
...
</bean>
Javaconfiguration:
Listing4.2
IgniteConfigurationcfg1=newIgniteConfiguration();
cfg1.setGridName("name1");
//Enableclientmode.
cfg1.setClientMode(true);
//StartIgnitenodeinclientmode
Igniteignite1=Ignition.start(cfg1);
Here is also a special type of logical node called a compute node in the Ignite cluster. A
computenodeisthenodethatusuallyparticipatesincomputingbusinesslogic.Basically,a
servernodethatcontainsdataisalsousedtoexecutecomputingtasks.