Automated Dynamic Error Analysis Methods for Optimization of Computer Arithmetic Systems

202 Pages·2015·1.63 MB·English

by  Michael Frechtling

8
OUR IDENTITY
LOGO
MASTERBRAND
LOGO
The University of Sydney logo consists of
PRIMARY LOGO CONFIGURATION Primary Logo
two elements: the shield and the University of
The primary logo is a principal
Sydney wordmark.
element of the University of
Both elements have been visually updated to create Sydney visual identity system.
It should be favoured and used,
a new, modern logo. The stylised shield and the
in this configuration, wherever
contemporary serif typeface reinforce our history
possible.
and origins and nod to the now.
The logo should be seen as a complete unit, with
the shield and wordmark always appearing together.
Only the logo artwork files can be used, including for
mono and mono-reverse applications. Logo artwork SECONDARY LOGO Secondary Logo - Stacked Secondary Logo - Horizontal
CONFIGURATIONS
files have been created for all logo uses, and must not
Stacked & Horizontal
be created.
To allow for flexibility in use
The logo should not be redrawn, digitally manipulated and application, two other logo
or altered. The following guidelines covering colour, treatments have been created:
stacked and horizontal. They
minimum size and clear space must be used when
should only be used where
applying the logo. message space and format
considerations demand it.
Stacked
The stacked logo should only be
used in vertically oriented, long
and thin applications.
Horizontal
The horizontal loSgoC HshOoOulLd OF ELECTRICAL & INFORMATION ENGINEERING
only be used in exaggerated
COMPUTER ENGINEERING LABORATORY
horizontal oriented spaces
and applications.
Automated Dynamic Error Analysis
Methods for Optimization of Computer
Arithmetic Systems
MichaelFRECHTLINGB.E.(Hons)
Athesissubmittedinfulfilmentof
therequirementsforthedegreeof
DoctorofPhilosophy
31stMarch2015
Formyparents
Abstract
Computerarithmeticisoneofthemoreimportanttopicswithincomputerscience
andengineering. Theearliestimplementationsofcomputersystemsweredesigned
toperformarithmeticoperationsandmostifnotalldigitalsystemswillberequired
toperformsomesortofarithmeticaspartoftheirnormaloperations. Thisreliance
on the arithmetic operations of computers means the accurate representation of
real numbers within digital systems is vital, and an understanding of how these
systemsareimplementedandtheirpossibledrawbacksisessentialinordertodesign
and implement modern high performance systems. At present the most widely
implementedsystemforcomputerarithmeticistheIEEE754FloatingPointsystem,
while this system is deemed to the be the best available implementation it has
severalfeaturesthatcanresultinseriouserrorsofcomputationifnotimplemented
correctly. Lack of understanding of these errors and their effects has lead to real
worlddisastersinthepastonseveraloccasions. Systemsforthedetectionofthese
errorsarehighlyimportantandfast,efficientandeasytouseimplementationsof
these detection systems is a high priority. Detection of floating point rounding
errorsnormallyrequiresrun-timeanalysisinordertobeeffective. Severalsystems
havebeenproposedfortheanalysisoffloatingpointarithmeticincludingInterval
Arithmetic,AffineArithmeticandMonteCarloArithmetic. Whilethesesystemshave
beenwellstudiedusingtheoreticalandsoftwarebasedapproaches,implementation
ofsystemsthatcanbeappliedtorealworldsituationshasbeenlimitedduetoissues
withimplementation,performanceandscalability. Themajorityofimplementations
havebeensoftwarebasedandhavenottakenadvantageoftheperformancegains
iv
associatedwithhardwareacceleratedcomputerarithmeticsystems. Thisisespecially
problematicwhenitisconsideredthatsystemsrequiringhighaccuracywilloften
requirehighperformance. Theaimofthisthesisandassociatedresearchistoincrease
understandingoferroranderroranalysismethodsthroughthedevelopmentofeasy
touseandeasytounderstandimplementationsofthesetechniques.
Acknowledgements
Iwouldliketoexpressmysincerethanksandappreciationtomysupervisorand
mentor,ProfessorPhilipH.W.Leong. Youradvice,encouragementanddirectionon
myresearch,mythesisandmycareerhavebeeninvaluableoverthelastfouryears
andIamextremelygratefultohavehadtheopportunitytoworkwithyou. Iwould
also like to thank my associate supervisor, Professor Craig Jin, whose advice and
assistancehas alsobeen invaluable. Iwould especiallylike tothank the pastand
presentmembersoftheComputerEngineeringLaboratoryandCARLab,inparticu-
larMr. NicholasFraser,Ms. CallaKlafas,Mr. DuncanMoss,Mr. StephenTridgell,
Dr. NicolasEpain,Dr. AengusMartin,Dr. AbhayaParthyandDr. AndrewWabnitz.
Yourabilitytoprovideassistance,adviceand,mostimportantly,distractionshave
helpedinmakingthelastfouryearssurvivable.
A special thanks to my family. Words cannot express how grateful I am to my
parents,brothersandextendedfamilyforallofthesacrificesthatyou’vemadeon
mybehalfandwhoseloveandsupporthavemadethisworkpossible. Iwouldalso
liketothankallofmyfriendswhosupportedmeinmywork,andencouragedme
tostrivetowardsmygoal.

Statement of Originality
Thisistocertifythattothebestofmyknowledge,thecontentofthisthesisismy
own work and contains no materials previously published or written by another
person, or substantial proportions of material which have been accepted for the
award of any other degree or diploma at the University of Sydney or any other
educationalinstitution,exceptwheredueacknowledgmentismadeinthethesis.
I certify that the intellectual content of this thesis is the product of my own work
andthatalltheassistancereceivedinpreparingthisthesisandsourceshavebeen
acknowledged,specifically:
• The research direction and subject were suggested by Professor Philip H.W.
Leong.
• Advice and editing assistance for the preparation of this thesis has been
providedbyProfessorPhilipH.W.Leong.
• Thecillysource-to-sourcecompileraspresentedinChapter4wasdeveloped
byProfessorPhilipH.W.Leong.
• MonteCarloarithmeticwasfirstproposedbyProfessorD.S.ParkeratUCLA[141].
Contents
Abstract iii
Acknowledgements v
StatementofOriginality vii
ListofFigures xiii
ListofTables xv
ListofAcronyms xvii
1 Introduction 1
1.1 Motivation&Aims . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.3 OrganizationoftheThesis . . . . . . . . . . . . . . . . . . . . . . . . . 4
2 Background 5
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2 ComputerArithmetic . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.2.1 BinaryNumberSystem&IntegerArithmetic . . . . . . . . . . 6
2.2.2 BasicOperations . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.2.3 SignedRepresentationandComplementOperations . . . . . 11
2.3 FixedPointArithmetic . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.3.1 FormatandBasicNotation . . . . . . . . . . . . . . . . . . . . 14
2.3.2 BasicOperators . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Contents ix
2.3.3 OverflowandPrecisionLoss . . . . . . . . . . . . . . . . . . . 18
2.4 FloatingPointArithmetic . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.4.1 NormalizedValues . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.4.2 Exact&InexactValues . . . . . . . . . . . . . . . . . . . . . . . 22
2.4.3 Rounding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.4.4 Overflow&Underflow . . . . . . . . . . . . . . . . . . . . . . 25
2.5 IEEE-754FloatingPointStandard . . . . . . . . . . . . . . . . . . . . . 26
2.5.1 BasicOperators . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.5.2 SpecialValues . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
2.5.3 Rounding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
2.5.4 Exceptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
2.6 ErrorAnalysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
2.6.1 ArithmeticError . . . . . . . . . . . . . . . . . . . . . . . . . . 38
2.6.2 NumericalStability . . . . . . . . . . . . . . . . . . . . . . . . . 38
2.6.3 Round-offError . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
2.6.4 CatastrophicCancellation . . . . . . . . . . . . . . . . . . . . . 41
2.6.5 Static&DynamicErrorAnalysisMethods . . . . . . . . . . . 42
2.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
3 MonteCarloArithmetic(MCA) 55
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
3.2 MonteCarloMethods . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
3.2.1 HistoryandDevelopment . . . . . . . . . . . . . . . . . . . . . 56
3.2.2 DefinitionandImplementation . . . . . . . . . . . . . . . . . . 57
3.2.3 SamplingMethods . . . . . . . . . . . . . . . . . . . . . . . . . 58
3.3 QuasiMonteCarloMethods . . . . . . . . . . . . . . . . . . . . . . . . 63
3.3.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
3.3.2 MeasuringDiscrepancy . . . . . . . . . . . . . . . . . . . . . . 64
3.3.3 Pseudo-Randomv. Quasi-Random . . . . . . . . . . . . . . . . 67
x Contents
3.3.4 EffectonRateofConvergence. . . . . . . . . . . . . . . . . . . 68
3.3.5 RandomizedQuasi-MonteCarloMethods . . . . . . . . . . . 69
3.4 MonteCarloArithmetic . . . . . . . . . . . . . . . . . . . . . . . . . . 70
3.4.1 ModellingInexactValues . . . . . . . . . . . . . . . . . . . . . 72
3.4.2 PrecisionBounding . . . . . . . . . . . . . . . . . . . . . . . . . 73
3.4.3 RandomRounding . . . . . . . . . . . . . . . . . . . . . . . . . 74
3.4.4 VirtualPrecisiont . . . . . . . . . . . . . . . . . . . . . . . . . . 78
3.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
4 MCALIB-AToolforAutomatedRoundingErrorAnalysis 81
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
4.2 MCALIBImplementation . . . . . . . . . . . . . . . . . . . . . . . . . 82
4.2.1 Source-to-SourceCompilation . . . . . . . . . . . . . . . . . . 82
4.2.2 LibraryImplementationusingMPFR . . . . . . . . . . . . . . 83
4.2.3 MCALIBFeatures&Workflow . . . . . . . . . . . . . . . . . . 85
4.3 AnalysisofMCAResults . . . . . . . . . . . . . . . . . . . . . . . . . . 89
4.3.1 LinearRegressionAnalysis . . . . . . . . . . . . . . . . . . . . 90
4.3.2 AssumptionofNormalityandConditionsonResults . . . . . 92
4.4 Testing&CaseStudies . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
4.4.1 ChebyshevPolynomials . . . . . . . . . . . . . . . . . . . . . . 92
4.4.2 SummationAlgorithm . . . . . . . . . . . . . . . . . . . . . . . 93
4.4.3 LinearAlgebra . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
4.4.4 L-BFGSOptimization . . . . . . . . . . . . . . . . . . . . . . . 96
4.5 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
4.5.1 ErrorDetectionandOptimizationofSampleAlgorithms . . . 98
4.5.2 Comparison of Single and Double Precision Floating Point
Formats . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
4.5.3 ComparisonofAlgorithmImplementations. . . . . . . . . . . 102
4.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104