Securing and Protecting Enterprise Networks via Data-driven Analytics and Application-aware SDN

118 Pages·2016·3.63 MB·English

by  Hesham Mekky

Securing and Protecting Enterprise Networks via
Data-driven Analytics and Application-aware SDN
A DISSERTATION
SUBMITTED TO THE FACULTY OF THE GRADUATE SCHOOL
OF THE UNIVERSITY OF MINNESOTA
BY
Hesham Mekky
IN PARTIAL FULFILLMENT OF THE REQUIREMENTS
FOR THE DEGREE OF
Doctor Of Philosophy
Prof. Zhi-Li Zhang
September, 2016
c Hesham Mekky 2016
�
ALL RIGHTS RESERVED
Acknowledgements
There are many people that have earned my gratitude for their contribution to my time
in graduate school.
First of all, I would like to thank my adviser, Professor Zhi-Li Zhang, for his con-
tinuous support, criticism, and guidance on conducting research and achieving personal
long-term career goals.
I am grateful to my collaborators Ruben Torres, Fang Hao, Sarit Mukherjee, T V
Lakshman, and Aziz Mohaisen. Their valuable feedback helped me turn my research
e↵orts into reality.
I thank my lab mates for collaborating with me on a number of exciting research
projects that greatly enriched my research experience. In particular, Gyan Ranjan,
Yang Zhang, Cheng Jin, Braulio Dumba, and Eman Ramadan.
I thank all my fellow graduate students working with Professors Zhi-Li Zhang, Tian
He, andDavidDufortheirvaluablefeedbackonmyresearchduringourgroupmeetings
and for creating and maintaining a friendly and intellectually creative environment.
I thank Professors Tian He, Abhishek Chandra, Andrew Odlyzko, David Du for
their support, feedback and advice as my PhD thesis (or oral) committee members.
Finally, I thank my wife, Sara, for her support and encouragement over the past five
years. I am also thankful to my parents for supporting me over the years.
i
Dedication
To my family.
ii
Abstract
The popularity of online services, such as social networks and online banking, has made
them a popular platform for attackers. Cybercriminals leverage them to spread mali-
cious software (malware) and steal personal information. In a cybercriminal operation,
miscreants infect their victims’ machines with malware that performs malicious activ-
ities. This occurs due to poor security measures implemented by enterprise networks,
and the complexity of network management tools. By studying existing malware distri-
bution networks and enterprise networks management tools, we aim to understand the
techniques used to infect victims such as drive-by downloads, study malware families
anddesignbetterdetectionmethodologies, andseeksolutionstowardsanimprovednet-
work management framework. Towards these goals, this thesis studies three orthogonal
problems aiming at addressing security and management problems in modern networks.
First, we study malware infections due to drive-by downloads using a large ISP
dataset. Weshowthatattackersemployredirectionswhichautomaticallyredirectusers’
requests through a series of intermediate websites, before landing on the final distri-
bution site. To detect these malicious redirections, we developed a machine learning
framework that relies on a distinctive set of features to label the malicious redirections
and block them.
Second, we study malware network traces for infected hosts in an enterprise network
using real malware traces, and we show that malware tra�c comes mixed up with legit-
imate user tra�c such as browsing tra�c. To improve malware detection, we developed
a novel system that decomposes the tra�c into separate components, and applies the
detection system to the suspected malware component only, and consequently improve
detection rates.
Third, we postulate native network functions within the Software-defined Network
(SDN) data plane, where the same logical controller controls both network services and
routing. This is enabled by extending Software-defined Networking to support stateful
flowhandlingbasedonhigherlayersinthepacketbeyondlayers2-4. Asaresult,network
functions (a.k.a middleboxes) can be chained on demand, directly on the data plane.
We present an implementation of this architecture based on Open vSwitch, and show
iii
that it enables popular network functions e↵ectively and addresses the management
problems in enterprise networks.
In summary, this thesis addresses these three closely related problems by: (1) pro-
tecting enterprise networks from drive-by downloads launched using redirections via a
data-drivenapproach; (2)detectingexistingmalwareactivityonthenetworkbydecom-
posing the end-host tra�c into a benign component and a suspected malware compo-
nent, then classifying the malware into its malware family; and (3) building a flexible
network architecture that enables managing network functions (e.g., such as systems in
(1) and (2) and others like firewalls and load balancers) within the data plane along
with the routing using a unified control plane.
iv
Contents
Acknowledgements i
Dedication ii
Abstract iii
List of Tables ix
List of Figures x
1 Introduction 1
1.1 Thesis Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 Outline and Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.3 Bibliographic Notes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2 Background and Motivation 5
2.1 Drive-by Download Attacks . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.2 Malware Families Classification . . . . . . . . . . . . . . . . . . . . . . . 8
2.3 Network Management and Middleboxes . . . . . . . . . . . . . . . . . . 8
3 Protection from drive-by download attacks 12
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
3.2 Main Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
3.3 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.4 Embedded Objects, HTTP Redirections, & Malware Exploits . . . . . . 16
3.5 Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
v
3.6 Browsing Activity Trees . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3.6.1 Tree Construction Algorithm . . . . . . . . . . . . . . . . . . . . 19
3.6.2 Tree Construction Assessment . . . . . . . . . . . . . . . . . . . 21
3.7 Identifying Malicious Redirections . . . . . . . . . . . . . . . . . . . . . 22
3.7.1 Systematic Study of Benign and Malicious Redirections . . . . . 22
3.7.2 Feature Selection and the Classifier . . . . . . . . . . . . . . . . . 24
3.8 System Prototype. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
3.9 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
3.9.1 Ground Truth. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
3.9.2 Classification Results. . . . . . . . . . . . . . . . . . . . . . . . . 31
3.9.3 False Positives Analysis . . . . . . . . . . . . . . . . . . . . . . . 32
3.9.4 Feature Importance . . . . . . . . . . . . . . . . . . . . . . . . . 34
3.10 Deployment & Integration with Existing Solutions . . . . . . . . . . . . 34
3.11 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
4 Detecting existing malware on the network 37
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
4.2 Main Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
4.3 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
4.4 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
4.4.1 Behavior-based Analysis . . . . . . . . . . . . . . . . . . . . . . . 41
4.4.2 ICA Primer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
4.4.3 Malware Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . 44
4.5 System Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
4.5.1 ICA Separation Module . . . . . . . . . . . . . . . . . . . . . . . 46
4.5.2 Classification Module . . . . . . . . . . . . . . . . . . . . . . . . 47
4.6 Features Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
4.6.1 Feature Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . 48
4.6.2 Feature Interpretation . . . . . . . . . . . . . . . . . . . . . . . . 50
4.6.3 Feature Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . 51
4.7 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
4.7.1 ICA Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
vi
4.7.2 Classifier Performance . . . . . . . . . . . . . . . . . . . . . . . . 54
4.8 Challenges and Limitations . . . . . . . . . . . . . . . . . . . . . . . . . 56
4.9 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
5 Managing Network Functions in Virtualized Data Centers 60
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
5.2 Main Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
5.3 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
5.4 System Overview and Challenges . . . . . . . . . . . . . . . . . . . . . . 63
5.5 System Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
5.5.1 Existing Open vSwitch Design . . . . . . . . . . . . . . . . . . . 67
5.5.2 Design Choices . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
5.5.3 Integrating with Open vSwitch flow processing . . . . . . . . . . 70
5.5.4 App Table and App Actions . . . . . . . . . . . . . . . . . . . . . 71
5.5.5 App Chaining and Execution . . . . . . . . . . . . . . . . . . . . 72
5.5.6 Flow Installation and Removal . . . . . . . . . . . . . . . . . . . 72
5.5.7 Applications State Management . . . . . . . . . . . . . . . . . . 73
5.5.8 An Example: Firewall and Load Balancer . . . . . . . . . . . . . 74
5.5.9 Loadable App Actions . . . . . . . . . . . . . . . . . . . . . . . . 75
5.6 Network Services Chaining . . . . . . . . . . . . . . . . . . . . . . . . . 76
5.7 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
5.7.1 Experimental Testbed Setup . . . . . . . . . . . . . . . . . . . . 79
5.7.2 Firewall Network Function . . . . . . . . . . . . . . . . . . . . . 79
5.7.3 Content-aware Server Selection Network Function . . . . . . . . 82
5.7.4 Loadable App-table Modules . . . . . . . . . . . . . . . . . . . . 86
5.8 Challenges and Open Questions . . . . . . . . . . . . . . . . . . . . . . . 86
5.9 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
6 Conclusions and Future Directions 89
6.1 Summary of Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . 89
6.2 Future Directions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
vii
Appendix A. Publications 103
A.1 Publications by Date . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
viii