Improving Accuracy of Software Performance Models on Multicore Platforms with Shared Caches

191 Pages·2012·4.68 MB·English

by  Vlastimil Babka

Charles University in Prague
Faculty of Mathematics and Physics
DOCTORAL THESIS
Vlastimil Babka
Improving Accuracy of Software
Performance Models on Multicore
Platforms with Shared Caches
Department of Distributed and Dependable Systems
Advisor: Doc. Ing. Petr T˚uma, Dr.
Study programme: Computer Science
Specialization: Software Systems
Prague 2012

Acknowledgments
It is my pleasant obligation to thank all those who supported me in my doc-
toralstudiesandintheresearchwhichresultedinthisthesis. Iamdeeplygrateful
to my advisor, Petr T˚uma for his help, guidance and co-authorship of all of the
included papers. I thank all my colleagues from the department for their contin-
uous feedback and fruitful discussion. In particular, my thanks go to the rest of
my co-authors and all fellows from the office room 205: Lubom´ır Bulej, Martin
Dˇecky´, Vojtˇech Horky´, Peter Libiˇc, Luk´aˇs Marek, Toma´ˇs Martinec and Andrej
Podzimek. I am also grateful to Frantiˇsek Pla´ˇsil for creating such an inspiring
research environment that our department, and the former research group, has
been.
ThisworkandtherelatedresearchwaspartiallysupportedbytheGrantAgen-
cy of the Czech Republic projects 201/05/H014, 201/09/H057 and 202/10/J04,
bytheCzechAcademyofSciencesproject1ET400300504,bytheMinistryofEdu-
cation of the Czech Republic project MSM0021620838, by the Charles University
institutional funding projects SVV-2010-261312, SVV-2011-263312, SVV-2012-
265312, and by the Q- ImPrESS research project by the European Union under
the ICT priority of the 7th Research Framework Programme.
Last but not least, I am in debt to my parents, my wife Kateˇrina and my son
Vojtˇech. Their endless support and patience made this work possible.
I hereby declare that I have authored this doctoral thesis on my own1, using only
the cited literature and other technical sources.
I understand that my work relates to the rights and obligations under the Act
No. 121/2000 Coll., the Copyright Act, as amended, in particular the fact that
the Charles University in Prague has the right to conclude a license agreement
on the use of this work as a school work pursuant to Section 60 paragraph 1 of
the Copyright Act.
In Prague, on ............ ........................
1The papers included in Part II have been written in cooperation with their respective
co-authors.
Annotations
Title
Improving Accuracy of Software Performance Models on Multicore Platforms
with Shared Caches
Author
Vlastimil Babka
e-mail: [email protected]ff.cuni.cz, phone: [phone]
Department
Department of Distributed and Dependable Systems
Faculty of Mathematics and Physics
Charles University in Prague, Czech Republic
Advisor
Doc. Ing. Petr T˚uma, Dr.
Department of Distributed and Dependable Systems
e-mail: [email protected]ff.cuni.cz, phone: [phone]
Abstract
The context of this work are performance models of software systems, which
are used for predicting performance of a system in its design phase. For
this purpose, performance models capture the explicit interactions of software
components that make up the system, and the resource demands of primitive
actions performed by the components. On contemporary hardware platforms,
the software components however interact also through implicit sharing of nu-
merous resources such as processor caches, which influence the performance
of the primitive actions. Implicit resource sharing is often omitted in perfor-
mance models, which affects their prediction accuracy.
In this work we introduce two methods for including resource sharing models
in performance models. Next, we propose an approximate resource sharing
model based on linear regression, and a detailed model for predicting per-
formance impact of cache sharing. The cache model is validated on a real
processor and its design is preceded by extensive experiments which investi-
gate the performance aspects of cache sharing. In addition, we introduce a
methodforrobustvalidationofperformancemodelsusingmanyautomatically
generated applications.
Keywords
performance models, resource sharing, cache models
Anotace
N´azev pr´ace
Vylepˇsen´ı pˇresnosti vy´konnostn´ıch model˚u software na v´ıcej´adrovy´ch plat-
forma´ch se sd´ıleny´mi cache
Autor
Vlastimil Babka
e-mail: [email protected]ff.cuni.cz, tel.: [phone]
Katedra
Katedra distribuovany´ch a spolehlivy´ch syst´em˚u
Matematicko-fyzik´aln´ı fakulta
Univerzita Karlova v Praze
Sˇkolitel
Doc. Ing. Petr T˚uma, Dr.
Katedra distribuovany´ch a spolehlivy´ch syst´em˚u
e-mail: [email protected]ff.cuni.cz, tel.: [phone]
Abstrakt:
Kontextem t´eto pr´ace jsou vy´konnostn´ı modely softwarovy´ch syst´em˚u, kter´e
slouˇz´ı pro predikci vy´konnosti syst´emu ve fa´zi jeho na´vrhu. K tomuto u´ˇcelu
potˇrebuj´ı vy´konnostn´ı modely popsat explicitn´ı interakce softwarovy´ch kom-
ponent, ze ktery´ch je syst´em sloˇzeny´, a vyuˇzit´ı prostˇredk˚u bˇehem primitivn´ıch
akc´ıvykona´vany´chkomponentami. Nasouˇcasny´chhardwarovy´chplatform´ach
ale softwarov´e komponenty interaguj´ı tak´e skrze implicitn´ı sd´ılen´ı prostˇredk˚u
jako jsou procesorov´e cache, coˇz ovlivnˇuje vy´kon prova´dˇeny´ch primitivn´ıch
akc´ı. Implicitn´ı sd´ılen´ı prostˇredk˚uˇcasto nen´ı zahrnuto ve vy´konnostn´ıch mod-
elech, coˇz negativnˇe ovlivnˇuje pˇresnost jejich predikc´ı.
Vt´etopr´acipˇredstavujemedvˇemetodyprozahrnut´ımodel˚usd´ılen´ıprostˇredk˚u
do vy´konnostn´ıch model˚u. D´ale navrhujeme zjednoduˇseny´ model sd´ılen´ı pros-
tˇredk˚u zaloˇzeny´ na line´arn´ı regresi, a detailn´ı model pro predikci zmˇen vy´konu
v d˚usledku sd´ılen´ı cache. Model sd´ılen´ı cache je ovˇeˇren na re´aln´em procesoru
a jeho na´vrhu pˇredcha´z´ı mnoˇzstv´ı experiment˚u, kter´e zkoumaj´ı vy´konnostn´ı
aspekty sd´ılen´ı cache. Pr´ace da´le navrhuje metodu pro robustn´ı ovˇeˇrov´an´ı
predikc´ıvy´konnostn´ıchmodel˚upomoc´ımnohaautomatickygenerovany´chsoft-
warovy´ch aplikac´ı.
Kl´ıˇcov´a slova
vy´konnostn´ı modely, sd´ılen´ı prostˇredk˚u, modely cache
Contents
1 Introduction 3
1.1 Performance Prediction of Software Systems . . . . . . . . . . . . 3
1.2 Implicit Resource Sharing . . . . . . . . . . . . . . . . . . . . . . 5
1.3 Memory Caches . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.4 Goals of the Thesis . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.5 Structure of the Thesis . . . . . . . . . . . . . . . . . . . . . . . . 10
I Overview of Contribution 11
2 Including Resource Sharing into Performance Prediction Models 13
2.1 Iterative Solution of Performance and Resource Models . . . . . . 15
2.2 Integrating Resource Models into Performance Model Solution Tools 18
3 Validating Performance Prediction Models on Many Applica-
tions 21
4 Approximating Impact of Sharing by Linear Models 27
5 Investigating Memory Caches of x86 Processors 31
5.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
5.2 Determining Undocumented Cache Parameters by Experiments . 31
5.3 Identifying and Quantifying the Effects of Cache Sharing . . . . . 33
5.3.1 Pipelined Composition . . . . . . . . . . . . . . . . . . . . 34
5.3.2 Parallel Composition . . . . . . . . . . . . . . . . . . . . . 35
5.3.3 Cache Miss Penalties in Cache Sharing Models . . . . . . . 35
5.3.4 Investigating Sharing Factors Other Than Cache Capacity
Competition . . . . . . . . . . . . . . . . . . . . . . . . . . 36
5.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
6 Modeling Cache Capacity Sharing Using Stack Distance Profiles 41
6.1 Choosing Appropriate Model Inputs . . . . . . . . . . . . . . . . . 41
6.2 On Existing Cache Sharing Models . . . . . . . . . . . . . . . . . 42
6.3 A New Cache Sharing Model . . . . . . . . . . . . . . . . . . . . . 43
6.4 Obtaining Model Inputs . . . . . . . . . . . . . . . . . . . . . . . 45
6.5 Model Validation and Stack Profile Collection Improvements . . . 46
6.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
II Collection of Papers 51
7 Resource Sharing in Performance Models 53
8 Validating Model-Driven Performance Predictions On Random
Software Systems 69
1
9 Can Linear Approximation Improve Performance Prediction? 87
10 Investigating Cache Parameters of x86 Family Processors 103
11 Timing Penalties Associated with Cache Sharing 125
12 When Misses Differ: Investigating Impact of Cache Misses on
Observed Performance 131
13 On The Accuracy of Cache Sharing Models 141
III Related Work and Conclusion 155
14 Related Work 157
14.1 Performance Models . . . . . . . . . . . . . . . . . . . . . . . . . 157
14.1.1 Our Work in the Context of Model-based Performance Pre-
diction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
14.1.2 Resource Sharing in Performance Models . . . . . . . . . . 158
14.2 Cache Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
14.2.1 Models for Isolated Workloads . . . . . . . . . . . . . . . . 162
14.2.2 CacheSharingModelsforInterleavedExecutionofMultiple
Workloads . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
14.2.3 Cache Sharing Models Focused on Scheduling Decisions . . 166
14.2.4 Cache Sharing Models for Parallel Execution of Multiple
Threads . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
15 Conclusion 173
15.1 Summary of Contribution . . . . . . . . . . . . . . . . . . . . . . 173
15.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174
Co-authored Publications and Tools 175
References 177
2
Chapter 1
Introduction
Performance is one of the important extra-functional properties of software sys-
tems. Besidesprovidingcorrectfunctionality, acommonrequirementforaninter-
active system is to process incoming requests of given rate fast enough to satisfy
the users performing them. Given two systems with the same functionality, a
system with better performance and scalability with respect to the number of
users will have a competitive advantage.
Given its importance, it is no surprise that evaluation of performance is a
thriving research field. In many decades, it has developed numerous methods and
toolsformeasuring,understandingandtuningperformanceofexistingsystems,as
well as simulation-based and analytical models predicting performance of systems
under design.
However, the increasing complexity of both hardware and software stack con-
tinues to bring new challenges for performance evaluation research. The exam-
ples of such challenges include multicore and hyperthreading processors, virtual
machines with just-in-time compilation and garbage collected heap, various mid-
dleware technologies and recently, hardware virtualization and cloud computing
platforms [16].
Typically, each such new technology requires improvements of existing eval-
uation techniques. For performance measurement, new metrics and tools are
needed to capture newly relevant performance properties and new benchmarks
have to be designed to compare performance of competing implementations. For
performance prediction, thorough understanding of the technologies and the ways
their usage affects performance of software systems is needed in order to improve
prediction accuracy of the existing models.
1.1 Performance Prediction of Software Systems
As we mentioned, one of the goals of performance evaluation is to predict per-
formance of a software system, e.g. its throughput, response time or resource
utilization, instead of actually measuring the system. This is naturally useful
when the system is not yet implemented and thus cannot be executed and mea-
sured, such as in its early design stages. A performance model of the system can
be used to verify that the system will meet its performance requirements or to
aid in various design decisions. Early indication of possible performance issues
makes them less difficult and expensive to fix than discovering them only after
the system is implemented [91].
Another scenario where prediction helps is when a system is implemented,
but subject to changes in its functional requirements, deployment or workload.
Then, the changes can be first applied to a model capturing the current state of
the system. The outcome of the changes on the performance requirements can
be then predicted before actually implementing the new functionality, changing
the deployment or before the workload becomes infeasible to cope with [100].
3
Recent research includes application of the change prediction to self-aware
systems, adding automatic decisions based on the predicted outcomes [68].
Performance prediction models
Performance prediction methods can be roughly divided into those based on ana-
lytical modeling and those based on simulation [62]. In both cases, a model of the
system is created that should capture the aspects of its structure and behavior
that are relevant to its performance. The level of detail in the model should be
high enough to capture all the important aspects, but at the same time coarse
enough to make the prediction practically feasible.
Analytical models are often based on probability theory and queueing theory
and can take the form of Markov chains, Layered Queueing Networks (LQN) [47]
or Queueing Petri Nets (QPN) [20] or other formalisms. The formalisms differ in
expressive power, which is often more or less limited by their solution tools, for
example by allowing only certain distributions of request inter-arrival times or
service times. On the other hand, they often have methods and tools for solving
the models analytically, i.e. calculating the predicted performance metrics of the
modelled system.
In comparison, simulation-based models offer theoretically unlimited expres-
sive power, but the time for estimating the predicted parameters tends to grow
with the complexity of the model. When analytical solution is infeasible, sim-
ulation can be used also to solve models created in analytical modeling for-
malisms [69].
One practical issue of performance prediction models preventing their wide-
spread use is that constructing the models properly requires rather expert perfor-
mance evaluation knowledge. Moreover, the formalisms used for creating perfor-
mance models are quite different from standard models used in software design
such as UML. A designer wishing to predict performance of a system during the
design process would have to create a separate performance model and keep both
the UML description and the performance model synchronized.
To bridge this gap, there is a large body of research devoted to model-based
performance prediction. In model-based approaches, the designer creates a de-
scription of the system (e.g., in UML) as usual, and adds annotations such as
service times of important methods or the rate of client requests. A performance
model is then created automatically by applying model transformation on the an-
notated UML description. After the performance model is solved, the results are
mappedbacktotheUMLlevel–thedesignerthusdoesnothavetounderstandor
manually maintain the performance model at all. Model-based approaches have
been surveyed in [19] and more recently in [70] focusing on component-based
approaches.
Validation of performance models
An important part of each publication proposing a new or improved performance
modeling formalism or methodology is its validation, which should demonstrate
that the approach is able to accurately predict performance of the type of systems
itisintendedfor. Thisistypicallydoneusingcasestudieswhereasoftwaresystem
4