Sequential and parallel algorithms and data structures

516 Pages·2019·3.626 MB·English

by  Sanders P

Peter Sanders
Kurt Mehlhorn
Martin Dietzfelbinger
Roman Dementiev
Sequential
and Parallel
Algorithms and
Data Structures
The Basic Toolbox
Sequential and Parallel
Algorithms and Data Structures
Peter Sanders • Kurt Mehlhorn
Martin Dietzfelbinger • Roman Dementiev
Sequential and Parallel
Algorithms and Data Structures
The Basic Toolbox
Peter Sanders Kurt Mehlhorn
Karlsruhe Institute of Technology Max Planck Institute for Informatics
Karlsruhe, Germany Saarbrücken, Germany
M artin Dietzfelbinger Roman Dementiev
Technische Universität Ilmenau Intel Deutschland GmbH
Ilmenau, Germany Feldkirchen, Germany
Intel material (including descriptions of algorithms and their implementations, text, figures,
and tables describing their experimental evaluation and experimental results on Intel processors,
and sample code) is reproduced by permission of Intel Corporation.

ISBN 978-3-030-25208-3 ISBN 978-3-030-25209-0 (eBook)
https://doi.org/10.1007/978-3-030-25209-0

© Springer Nature Switzerland AG 2019
This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of
the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations,
recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or
information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar
methodology now known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this
publication does not imply, even in the absence of a specific statement, that such names are exempt
from the relevant protective laws and regulations and therefore free for general use.
The publisher, the authors, and the editors are safe to assume that the advice and information in this
book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors
or the editors give a warranty, express or implied, with respect to the material contained herein or for
any errors or omissions that may have been made. The publisher remains neutral with regard to
jurisdictional claims in published maps and institutional affiliations.

This Springer imprint is published by the registered company Springer Nature Switzerland AG
The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland
To all algorithmicists
Preface
An algorithmis a preciseand unambiguousrecipefor solvinga classof problems.
If formulated as programs, they can be executed by a machine. Algorithms are at
the heart of every nontrivial computer application. Therefore, every computer sci-
entistandeveryprofessionalprogrammershouldknowaboutthebasic algorithmic
toolbox:structures thatallow efficient organizationand retrievalof data, key algo-
rithmsforproblemsongraphs,andgenerictechniquesformodeling,understanding,
andsolvingalgorithmicproblems.Thisbookisa conciseintroductiontothisbasic
toolbox,intendedforstudentsandprofessionalsfamiliarwithprogrammingandba-
sic mathematicallanguage.Althoughthe presentationisconcise,wehaveincluded
manyexamples,pictures,informalexplanations,andexercises,andsomelinkageto
the real world.The bookcoversthe materialtypicallytaughtin undergraduateand
first-yeargraduatecoursesonalgorithms.
Mostchaptershavethesamebasicstructure.Webeginbydiscussingaproblem
asitoccursinareal-lifesituation.Weillustratethemostimportantapplicationsand
thenintroducesimplesolutionsasinformallyaspossibleandasformallyasneces-
sary toreallyunderstandtheissuesathand.Whenwemovetomoreadvancedand
optionalissues,thisapproachgraduallyleadstoamoremathematicaltreatment,in-
cluding theorems and proofs. Thus, the book should work for readers with a wide
range of mathematical expertise. There are also advanced sections (marked with a
*) where we recommend that readers should skip them on first reading. Exercises
provideadditionalexamples,alternativeapproachesandopportunitiestothinkabout
theproblems.Itishighlyrecommendedtotakealookattheexercisesevenifthere
isnotimetosolvethemduringthefirstreading.Inordertobeabletoconcentrateon
ideasratherthanprogrammingdetails,weusepictures,words,andhigh-levelpseu-
docodetoexplainouralgorithms.Asection“ImplementationNotes”linkstheseab-
stractideastoclean,efficientimplementationsinrealprogramminglanguagessuch
asC++andJava.Eachchapterendswithasectiononfurtherfindingsthatprovides
aglimpseatthestateoftheart,generalizations,andadvancedsolutions.
Thefirst edition ofthis booktreatedonly sequentialalgorithmsand data struc-
tures. Today,almosteverycomputer,be it a desktop,a notebook,or a smartphone,
has multiple cores, and sequential machines have become an exotic species. The
viii Preface
reasonforthischangeis thatsequentialprocessorshaveceased togetproportional
performanceimprovementsfromincreasedcircuitcomplexity.Althoughthenumber
of transistorsin an integratedcircuit(still) doubleseverytwo years(Moore’slaw),
the only reasonable way to use this transistor budget is to put multiple processor
coresonachip.Theconsequenceisthatnowadayseveryperformance-criticalappli-
cationhastobeparallelized.Moreover,bigdata–theexplosionofdatasetsizesin
manyapplications–hasproducedanenormousdemandforalgorithmsthatscaleto
alargenumberofprocessors.Thisparadigmshifthasprofoundeffectsonteaching
algorithms.Parallelalgorithmsarenolongeraspecializedtopicreservedforasmall
percentageofstudents.Rather,everystudentneedssomeexposuretoparallelalgo-
rithms,andparallelsolutionparadigmsneedtobetaughtearlyon.Asaconsequence,
parallelalgorithmsshouldbeintegratedtightlyandearlyintoalgorithmscourses.We
therefore decided to include parallel algorithms in the second edition of the book.
Each chapter now has some sections on parallel algorithms. The goals remain the
same as for the first edition: a careful balance between simplicity and efficiency,
between theory and practice, and between classical results and the forefrontof re-
search.We use a slightlydifferentstyle forthesectionson parallelcomputing.We
includeconcreteprogrammingexamplesbecauseparallelprogrammingisstillmore
difficultthansequentialprogramming(theprogramsareavailableatgithub.com/
basic-toolbox-sample-code/basic-toolbox-sample-code/).We
also referenceoriginalwork directlyin the textinstead of in the sectionon history
becausetheparallelpartisclosertocurrentresearch.
Algorithmicsisamodernandactiveareaofcomputerscience,evenatthelevel
of the basic toolbox. We have made sure that we present algorithms in a modern
way, including explicitly formulated invariants. We also discuss important further
aspects,suchasalgorithmengineering,memoryhierarchies,algorithmlibraries,and
certifyingalgorithms.
Wehavechosentoarrangemostofthematerialbyproblemdomainandnotby
solutiontechnique.Thechapteronoptimizationtechniquesisanexception.Wefind
thatan organizationby problemdomainallowsa moreconcise presentation.How-
ever,itisalsoimportantthatreadersandstudentsobtainagoodgraspoftheavailable
techniques. Therefore, we have structured the optimization chapter by techniques,
and an extensive index providescross-referencesbetween differentapplicationsof
thesametechnique.Boldpagenumbersintheindexindicatethepageswherecon-
ceptsaredefined.
Thisbookcanbeusedinmultiplewaysinteaching.Wehaveusedthefirstedition
ofthebookandadraftversionofthesecondeditioninundergraduateandgraduate
coursesonalgorithmics.Inafirstyearundergraduatecourse,weconcentratedonthe
sequentialpartofthe book.Ina secondalgorithmscourseatan advancedbachelor
levelormaster’slevelandwithstudentswithsomeexperienceinparallelprogram-
ming we made most of the sequential part of the book a prerequisite and concen-
trated on the more advancedmaterial such as the starred sections and the parts on
externalmemoryandparallelalgorithms.Ifa firstalgorithmscourseis taughtlater
intheundergraduatecurriculum,thebookcanbeusedtoteachparallelandsequen-
tialalgorithmsinanintegratedway.Althoughwehaveincludedsomematerialabout
Preface ix
parallelprogrammingandseveralconcreteprogrammingexamples,theparallelpart
of the book works best for readers who already have some backgroundin parallel
programming.Anotherapproachistousethebooktoprovideconcretealgorithmic
contentfora parallelprogrammingcoursethatusesanotherbookforthe program-
ming part. Last but notleast, the book should be usefulfor independentstudy and
toprofessionalswhohaveabasicknowledgeofalgorithmsbutlessexperiencewith
parallelism.
Followusonanexcitingtourthroughtheworldofalgorithms.
Ilmenau,Saarbrücken,Karlsruhe,Heidelberg MartinDietzfelbinger
August2018 KurtMehlhorn
PeterSanders
RomanDementiev
Afirsteditionofthisbookwaspublishedin2008.Sincethenthebookhasbeen
translatedintoChinese,Greek,Japanese,andGerman.MartinDietzfelbingertrans-
latedthebookintoGerman.Actually,hedidmuchmorethanatranslation.Hethor-
oughlyrevisedthebookandimprovedthepresentationatmanyplaces.Healsocor-
rectedanumberofmistakes.Thus,thebookgainedthroughthetranslation,andwe
decidedtomaketheGermaneditionthereferenceforanyfutureeditions.Itisonly
natural that we asked Martin to become an author of the German edition and any
futureeditionsofthebook.
Saarbrücken,Karlsruhe KurtMehlhorn
March2014 PeterSanders
SoonafterthepublicationoftheGermanedition,westartedworkingonthere-
visedEnglishedition.Wedecidedtoexpandthebookintotheparallelworldforthe
reasons indicated in the preface. Twenty years ago, parallel machines were exotic,
nowadays,sequentialmachinesareexotic.However,theparallelworldismuchmore
diverseandcomplexthanthesequentialworld,andthereforealgorithm-engineering
issues becomemoreimportant.We concludedthatwe hadto goallthe wayto im-
plementationsandexperimentalevaluationsforsomeoftheparallelalgorithms.We
invitedRomanDementievto workwithusonthealgorithmengineeringaspectsof
parallelcomputing.RomanreceivedhisPhDin2006forathesison“AlgorithmEn-
gineeringforLargeDataSets”.HenowworksforIntel,whereheisresponsiblefor
performanceengineeringofamajordatabasesystem.
Ilmenau,Saarbrücken,Karlsruhe MartinDietzfelbinger
November2017 KurtMehlhorn
PeterSanders
Contents
1 Appetizer:IntegerArithmetic ................................... 1
1.1 Addition................................................. 2
1.2 Multiplication:TheSchoolMethod .......................... 6
1.3 ResultChecking .......................................... 10
1.4 ARecursiveVersionoftheSchoolMethod .................... 11
1.5 KaratsubaMultiplication ................................... 13
1.6 ParallelMultiplication ..................................... 15
1.7 AlgorithmEngineering..................................... 17
1.8 ThePrograms ............................................ 18
1.9 ProofsofLemma1.7andTheorem1.9 ....................... 21
1.10 ImplementationNotes...................................... 23
1.11 HistoricalNotesandFurtherFindings ........................ 23
2 Introduction................................................... 25
2.1 AsymptoticNotation....................................... 26
2.2 TheSequentialMachineModel.............................. 29
2.3 Pseudocode .............................................. 32
2.4 ParallelMachineModels ................................... 38
2.5 ParallelPseudocode ....................................... 44
2.6 DesigningCorrectAlgorithmsandPrograms .................. 46
2.7 AnExample–BinarySearch................................ 49
2.8 BasicAlgorithmAnalysis .................................. 52
2.9 Average-CaseAnalysis..................................... 57
2.10 Parallel-AlgorithmAnalysis ................................ 62
2.11 RandomizedAlgorithms.................................... 63
2.12 Graphs .................................................. 68
2.13 PandNP ................................................ 73
2.14 ImplementationNotes...................................... 77
2.15 HistoricalNotesandFurtherFindings ........................ 79
xii Contents
3 RepresentingSequencesbyArraysandLinkedLists ............... 81
3.1 ProcessingArraysinParallel................................ 82
3.2 LinkedLists.............................................. 86
3.3 ProcessingLinkedListsinParallel........................... 91
3.4 UnboundedArrays ........................................ 97
3.5 *AmortizedAnalysis ...................................... 102
3.6 StacksandQueues ........................................ 106
3.7 ParallelQueue-LikeDataStructures.......................... 109
3.8 ListsversusArrays ........................................ 113
3.9 ImplementationNotes...................................... 114
3.10 HistoricalNotesandFurtherFindings ........................ 116
4 HashTablesandAssociativeArrays.............................. 117
4.1 HashingwithChaining..................................... 120
4.2 UniversalHashing......................................... 122
4.3 HashingwithLinearProbing................................ 128
4.4 ChainingversusLinearProbing ............................. 130
4.5 *PerfectHashing.......................................... 131
4.6 ParallelHashing .......................................... 134
4.7 ImplementationNotes...................................... 147
4.8 HistoricalNotesandFurtherFindings ........................ 149
5 SortingandSelection ........................................... 153
5.1 SimpleSorters............................................ 156
5.2 Simple,Fast,andInefficientParallelSorting................... 158
5.3 Mergesort–anO(nlogn)SortingAlgorithm .................. 160
5.4 ParallelMergesort......................................... 162
5.5 ALowerBound........................................... 165
5.6 Quicksort ................................................ 168
5.7 ParallelQuicksort ......................................... 173
5.8 Selection................................................. 178
5.9 ParallelSelection.......................................... 180
5.10 BreakingtheLowerBound ................................. 182
5.11 *ParallelBucketSortandRadixSort ......................... 186
5.12 *ExternalSorting ......................................... 187
5.13 ParallelSampleSortwithImplementations.................... 191
5.14 *ParallelMultiwayMergesort............................... 201
5.15 ParallelSortingwithLogarithmicLatency..................... 205
5.16 ImplementationNotes...................................... 207
5.17 HistoricalNotesandFurtherFindings ........................ 208