Data Mining Academic Emails to Model Employee Behaviors and Analyze Organizational Structure

100 Pages·2016·2.36 MB·English

Data Mining Academic Emails to Model Employee Behaviors and
Analyze Organizational Structure
KaylaM.Straub
ThesissubmittedtotheFacultyofthe
VirginiaPolytechnicInstituteandStateUniversity
inpartialfulfillmentoftherequirementsforthedegreeof
MasterofScience
in
ElectricalEngineering
RobertW.McGwier,Chair
A.A.(Louis)Beex
R.MichaelBuehrer
BertHuang
March25,2016
Blacksburg,Virginia
Keywords: Dataanalytics,machinelearning,socialcomputing
Copyright2016,KaylaM.Straub
Data Mining Academic Emails to Model Employee Behaviors and Analyze
Organizational Structure
KaylaM.Straub
(ABSTRACT)
Email correspondence has become the predominant method of communication for businesses. If
not for the inherent privacy concerns, this electronically searchable data could be used to better
understandhowemployeesinteract. AftertheEnrondatasetwasmadeavailable,researcherswere
abletoprovidegreatinsightintoemployeebehaviorsbasedontheavailabledatadespitethemany
challengeswiththatdataset. Theworkinthisthesisdemonstratesasuiteofmethodsappliedtoan
appropriately anonymized academic email dataset created from volunteers’ email metadata. This
new dataset, from an internal email server, is first used to validate feature extraction and machine
learningalgorithmsinordertogenerateinsightintotheinteractionswithinthecenter. Basedsolely
on email metadata, a random forest approach models behavior patterns and predicts employee job
titles with 96% accuracy. This result represents classifier performance not only on participants
in the study but also on other members of the center who were connected to participants through
email. Furthermore, the data revealed relationships not present in the center’s formal operating
structure. The culmination of this work is an organic organizational chart, which contains a fuller
understanding of the center’s internal structure than can be found in the official organizational
chart.
Data Mining Academic Emails to Model Employee Behaviors and Analyze
Organizational Structure
KaylaM.Straub
(GENERALAUDIENCEABSTRACT)
Emailcorrespondencehasbecomethepredominantmethodofcommunicationforbusinessesbutis
seldomthesubjectofresearchduetotheinherentprivacyconcerns. Thisworkdemonstratesasuite
ofmethodsappliedtoanacademicemaildatasetcreatedfromvolunteers’emailmetadatainorder
to gain a better understanding of office email interactions. The metadata involved contains details
about an email such as the timestamp, length of text, and involved email addresses. However, all
email text was omitted to protect participant privacy. Meaningful behavior patterns are extracted
from this dataset in the form of statistical features, which are used in machine learning algorithms
to generate insight into the interactions within the center. The algorithm predicted employee job
titleswith96%accuracyincludingparticipantsinthestudyaswellasothermembersofthecenter
who were connected to participants through email. The data revealed relationships not present in
the center’s formal operating structure. The culmination of this work is an organic organizational
chart, which contains a fuller understanding of the center’s internal structure than can be found
in the official organizational chart. These procedures can be applied to corporate analyses to aid
in restructurings or mergers. Beyond this, these methods may be applied to any communications
systemtoidentifyanomalousbehavior,leadershipstructures,orcliques.
Acknowledgments
IwanttoacknowledgethemanypeoplewhohelpedmewithmygraduateworkatVirginiaTechand
made this work possible. Thank you to the members of my committee: Dr. Robert W. McGwier,
Dr. Louis Beex, Dr. Michael Buehrer, and Dr. Bert Huang for your guidance. In particular, I
wouldliketosincerelythankDr. McGwierforhisvisionandsupportthroughoutthisresearch.
I want to express my gratitude to the faculty, staff, and students of the Hume Center for all of
your help and for providing me with the opportunities to work on such interesting research. I am
especially thankful for Dr. Joseph Ernst and Dr. William C. Headley for their invaluable insight
andassistance.
I would also like to thank my family and friends for their support throughout my college career. I
wanttospecificallythankmyhusbandBenforhisunwaveringsupportandforalwayspushingme
tobemybest.
Finally,IthankGodforalltheblessingsinmylifeandforgettingmetowhereIamtoday.
iv
Contents
ListofFigures viii
ListofTables x
1 Introduction 1
1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 ProblemOverview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.4 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.5 Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2 PriorWork 6
2.1 EmailAnalysisOverview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.1.1 CommonlyUsedEmailDatasets . . . . . . . . . . . . . . . . . . . . . . . 7
2.1.2 PriorEmailAnalysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.2 MachineLearningOverview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.2.1 SubfieldsofMachineLearning . . . . . . . . . . . . . . . . . . . . . . . . 12
2.2.2 RandomForests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.2.3 OccupationClassification . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
3 EmailDatasetandFeatureExtraction 16
v
3.1 DataCollection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3.2 DatasetDescriptionandStatistics . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3.3 Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
3.3.1 Traffic-BasedFeatures . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
3.3.2 SocialNetworkFeatures . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
4 AlgorithmDesign 42
4.1 AlgorithmSelection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
4.2 AlgorithmDescription . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
4.3 FeatureSelection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
5 PerformanceAnalysis 51
5.1 ClassificationResults . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
5.2 Leave-One-OutCross-Validation . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
5.3 HierarchyAnalysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
6 FutureWork 63
6.1 ImprovingLOOCVAccuracy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
6.2 AlgorithmsforProcessingLargerData . . . . . . . . . . . . . . . . . . . . . . . . 66
6.2.1 DeepBeliefNetworks . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
6.2.2 CollectiveClassification . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
7 Conclusions 72
8 Bibliography 74
vi
AppendixA:ExampleEmail 84
AppendixB:FullFeatureList 88
vii
List of Figures
3.1 Examplesubjecthash . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.2 Centeremailsovertime . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
3.3 Datasetclassdistribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
3.4 Emailsperpersonperjobtitle . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
3.5 Forwardedemailsreceivedhistogram . . . . . . . . . . . . . . . . . . . . . . . . . 28
3.6 Thesocialnetworkofthecenter . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
3.7 Thedatasetrepresentedasanadjacencymatrix . . . . . . . . . . . . . . . . . . . 31
3.8 Typesofemailsemployeessenttothemselves . . . . . . . . . . . . . . . . . . . . 32
3.9 Histogramofthehubsfeatureforthepartialgraph . . . . . . . . . . . . . . . . . . 40
4.1 Algorithmselectionanalysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
4.2 Examplerandomtree . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
4.3 Randomforestpredictionprocess . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
5.1 Jobtitleclassificationresults . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
5.2 Jobtitlepredictionconfusionmatrix . . . . . . . . . . . . . . . . . . . . . . . . . 54
5.3 Accuracyofclassificationovertime . . . . . . . . . . . . . . . . . . . . . . . . . 55
5.4 Predictionaccuracycomparedtonumberoffeatures . . . . . . . . . . . . . . . . . 56
viii
5.5 Leave-one-outcross-validationresults . . . . . . . . . . . . . . . . . . . . . . . . 57
5.6 Generatedorganicorganizationchartofthecenter . . . . . . . . . . . . . . . . . . 60
5.7 Projectmanageranddirectorpredictionresults . . . . . . . . . . . . . . . . . . . . 62
6.1 Effectsofmoredataonpredictionaccuracy . . . . . . . . . . . . . . . . . . . . . 65
6.2 Effectsofmorepeopleonpredictionaccuracy . . . . . . . . . . . . . . . . . . . . 66
6.3 TheComponentsofaDeepBeliefNetwork . . . . . . . . . . . . . . . . . . . . . 67
ix
List of Tables
3.1 AcomparisonbetweentheinternaldatasetandtheEnronemailcorpus. . . . . . . . 20
4.1 Top20featuresrankedbythemutualinformation. . . . . . . . . . . . . . . . . . . 50
x