Alleviating Virtualization Bottlenecks

139 Pages·2014·1.63 MB·English

Alleviating Virtualization
Bottlenecks
Nadav Amit
Technion - Computer Science Department - Ph.D. Thesis PHD-2014-08 - 2014
Technion - Computer Science Department - Ph.D. Thesis PHD-2014-08 - 2014
Alleviating Virtualization
Bottlenecks
Research Thesis
Submitted in partial fulfillment of the requirements
for the degree of Doctor of Philosophy
Nadav Amit
Submitted to the Senate
of the Technion — Israel Institute of Technology
Tamuz 5774 Haifa July 2014
Technion - Computer Science Department - Ph.D. Thesis PHD-2014-08 - 2014
Technion - Computer Science Department - Ph.D. Thesis PHD-2014-08 - 2014
ThisresearchwascarriedoutunderthesupervisionofProf.AssafSchusterandProf.Dan
Tsafrir, in the Faculty of Computer Science.
Some results in this thesis have been published as articles by the author and research
collaborators in conferences and journals during the course of the author’s doctoral
research period, the most up-to-date versions of which being:
NadavAmit,MuliBen-Yehuda,DanTsafrir,andAssafSchuster. vIOMMU:efficientIOMMU
emulation. In USENIX Annual Technical Conference (ATC), 2011.
Nadav Amit, Dan Tsafrir, and Assaf Schuster. VSWAPPER: A memory swapper for
virtualized environments. In ACM Architectural Support for Programming Languages &
Operating Systems (ASPLOS), pages 349–366, 2014.
Abel Gordon, Nadav Amit, Nadav Har’El, Muli Ben-Yehuda, Alex Landau, Assaf Schuster,
and Dan Tsafrir. ELI: Bare-metal performance for I/O virtualization. In ACM Architectural
Support for Programming Languages & Operating Systems (ASPLOS), pages 411–422, 2012.
First two authors equally contributed.
Acknowledgements
I thank my advisors, Assaf Schuster and Dan Tsafrir, for their academic supervision
and inspiration. Through your guidance I learned how to separate the wheat from the
chaff in academic research. Your contribution to this work is invaluable. It has been a
privilege to work with you.
I am grateful to Muli Ben-Yehuda for bringing interesting problems to study and
guiding me in my first academic steps. I thank my co-authors Nadav Har’el, Abel
Gordon and Alex Landau for their important role in the research, and thank Ilya
Kravetz for his technical support. I also thank Michael Factor and Carl Waldspurger
for reviewing the studies and delivering valuable feedback.
During my studies, I got support from my colleagues: Mickey Gabel, Orna Agmon
Ben-Yehuda, Omer Peleg, Ilya Lesokhin, Moshe Malka, Eitan Rosenfeld, Eran Gilad,
GalaYadgar, AdiOmariIgorSmolyarandMichaelVasiliev. Yoursupportandfeedbacks
were useful throughout my studies.
Last but not least, I thank my parents Noah and Tamar Amit, and especially my
spouse Mika Moran. Your patience, understanding and support enabled me to conduct
this research.
The generous financial support of the Hasso Plattner Institue, IBM cooperation and
the Technion if gratefully acknowledged.
Technion - Computer Science Department - Ph.D. Thesis PHD-2014-08 - 2014
Technion - Computer Science Department - Ph.D. Thesis PHD-2014-08 - 2014
Contents
List of Figures
List of Tables
Abstract 1
Abbreviations and Notations 3
1 Introduction 7
1.1 Overheads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.1.1 Architecture Perspective . . . . . . . . . . . . . . . . . . . . . . . 8
1.1.2 High-Level Perspective . . . . . . . . . . . . . . . . . . . . . . . . 9
1.2 Achieving Transparency . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.2.1 Cooperative Transparency . . . . . . . . . . . . . . . . . . . . . . 10
1.2.2 Non-Cooperative Transparency . . . . . . . . . . . . . . . . . . . 11
1.3 Research Goal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2 vIOMMU 17
2.1 Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.2 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.2.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.2.2 Contributions and Preview of Results . . . . . . . . . . . . . . . 19
2.3 Samecore IOMMU Emulation . . . . . . . . . . . . . . . . . . . . . . . . 21
2.4 IOMMU Mapping Strategies . . . . . . . . . . . . . . . . . . . . . . . . 23
2.4.1 Approximate Shared Mappings . . . . . . . . . . . . . . . . . . . 23
2.4.2 Asynchronous Invalidations . . . . . . . . . . . . . . . . . . . . . 24
2.4.3 Deferred Invalidation . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.4.4 Optimistic Teardown . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.5 Sidecore IOMMU Emulation . . . . . . . . . . . . . . . . . . . . . . . . 26
2.5.1 Risk and Protection Types . . . . . . . . . . . . . . . . . . . . . 27
2.5.2 Quantifying Risk . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.6 Performance Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
2.6.1 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
Technion - Computer Science Department - Ph.D. Thesis PHD-2014-08 - 2014
2.6.2 Overhead of (Un)mapping . . . . . . . . . . . . . . . . . . . . . . 33
2.6.3 Benchmark Results . . . . . . . . . . . . . . . . . . . . . . . . . . 33
2.6.4 Sidecore Scalability and Power-Efficiency . . . . . . . . . . . . . 36
2.7 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
2.8 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
3 ELI 41
3.1 Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
3.2 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
3.3 Motivation and Related Work . . . . . . . . . . . . . . . . . . . . . . . . 44
3.3.1 Generic Interrupt Handling Approaches . . . . . . . . . . . . . . 44
3.3.2 Virtualization-Specific Approaches . . . . . . . . . . . . . . . . . 45
3.4 x86 Interrupt Handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
3.4.1 Interrupts in Bare-Metal Environments . . . . . . . . . . . . . . 46
3.4.2 Interrupts in Virtual Environments . . . . . . . . . . . . . . . . . 47
3.4.3 Interrupts from Assigned Devices . . . . . . . . . . . . . . . . . . 48
3.5 ELI: Design and Implementation . . . . . . . . . . . . . . . . . . . . . . 49
3.5.1 Exitless Interrupt Delivery . . . . . . . . . . . . . . . . . . . . . 49
3.5.2 Placing the Shadow IDT . . . . . . . . . . . . . . . . . . . . . . . 51
3.5.3 Configuring Guest and Host Vectors . . . . . . . . . . . . . . . . 52
3.5.4 Exitless Interrupt Completion . . . . . . . . . . . . . . . . . . . . 52
3.5.5 Multiprocessor Environments . . . . . . . . . . . . . . . . . . . . 53
3.6 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
3.6.1 Methodology and Experimental Setup . . . . . . . . . . . . . . . 53
3.6.2 Throughput . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
3.6.3 Execution Breakdown . . . . . . . . . . . . . . . . . . . . . . . . 57
3.6.4 Impact of Interrupt Rate . . . . . . . . . . . . . . . . . . . . . . 58
3.6.5 Latency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
3.7 Security and Isolation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
3.7.1 Threat Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
3.7.2 Protection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
3.8 Architectural Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
3.9 Applicability and Future Work . . . . . . . . . . . . . . . . . . . . . . . 63
3.10 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
4 VSWAPPER 65
4.1 Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
4.2 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
4.3 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
4.3.1 The Benefit of Ballooning . . . . . . . . . . . . . . . . . . . . . . 68
4.3.2 Ballooning is Not a Complete Solution . . . . . . . . . . . . . . . 69
Technion - Computer Science Department - Ph.D. Thesis PHD-2014-08 - 2014
4.3.3 Ballooning Takes Time. . . . . . . . . . . . . . . . . . . . . . . . 71
4.3.4 The Case for Unmodified Guests . . . . . . . . . . . . . . . . . . 73
4.4 Baseline Swapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
4.4.1 Demonstration . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
4.5 Design and Implementation . . . . . . . . . . . . . . . . . . . . . . . . . 79
4.5.1 The Swap Mapper . . . . . . . . . . . . . . . . . . . . . . . . . . 80
4.5.2 The False Reads Preventer . . . . . . . . . . . . . . . . . . . . . 83
4.6 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
4.6.1 Controlled Memory Assignment . . . . . . . . . . . . . . . . . . . 85
4.6.2 Dynamic Memory Assignment . . . . . . . . . . . . . . . . . . . 88
4.6.3 Overheads and Limitations . . . . . . . . . . . . . . . . . . . . . 89
4.6.4 Non-Linux Guests and Hosts . . . . . . . . . . . . . . . . . . . . 90
4.7 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
4.8 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
4.9 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
4.10 Availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
5 Conclusion and open questions 95
Hebrew Abstract i
Technion - Computer Science Department - Ph.D. Thesis PHD-2014-08 - 2014
Technion - Computer Science Department - Ph.D. Thesis PHD-2014-08 - 2014