Transforming Time Series for Efficient and Accurate Classification

178 Pages·2017·3.6 MB·English

2018 01
PhD-FSTC- -
Faculté des Sciences, de la Technologie et de la Communication
DISSERTATION
11 01 2018
Defense held on / / in Luxembourg
to obtain the degree of
DOCTEUR DE L’UNIVERSITÉ DU LUXEMBOURG
EN INFORMATIQUE
by
Daoyuan LI
BornonAugust5th,1987inHubei,China
TRANSFORMING TIME SERIES
FOR
EFFICIENT AND ACCURATE CLASSIFICATION
issertation defense committee
D
acques lein
Dr. J K , Chairman
SeniorResearchScientist,UniversitéduLuxembourg
egawendé issyandé
Dr. T F. B , Vice Chairman
ResearchScientist,UniversitéduLuxembourg
ves e raon
Dr. Y L T , Dissertation Supervisor
Professor,UniversitéduLuxembourg
essica in
Dr. J L
AssociateProfessor,GeorgeMasonUniversity
atthieu eist
Dr. M G
Professor,UniversitédeLorraine

ABSTRACT
Time series data refer to sequences of data that are ordered either temporally, spa-
tially or in another defined order. They can be frequently found in a variety of
domains, including financial data analysis, medical and health monitoring and
industrial automation applications. Due to their abundance and wide application
scenarios, there has been an increasing need for efficient machine learning algo-
rithms to extract information and build knowledge from these data. One of the
majortasksintimeseriesminingis timeseriesclassification(TSC),whichconsists
ofapplyingalearningalgorithmonlabeleddatatotrainamodelthatwillthenbe
used to predict the classes of samples from an unlabeled data set. Due to the se-
quential characteristic of time series data, state-of-the-art classification algorithms
(such as SVM and Random Forest) that performs well for generic data are usually
notsuitableforTSC.InordertoimprovetheperformanceofTSCtasks,thisdisser-
tationproposesdifferentmethodstotransformtimeseriesdataforabetterfeature
extraction process as well as novel algorithms to achieve better classification per-
formance in terms of computation efficiency and classification accuracy.
In the first part of this dissertation, we conduct a large scale empirical study
that takes advantage of discrete wavelet transform (DWT) for time series dimen-
sionality reduction. We first transform real-valued time series data using different
1
families of DWT. Then we apply dynamic time warping (DTW)-based NN clas-
39
sification on datasets and find out that existing DWT-based lossy compression
approaches can help to overcome the challenges of storage and computation time.
Furthermore,weprovideassurancestopractitionersbyempiricallyshowing,with
various datasets and with several DWT approaches, that TSC algorithms yield
similaraccuracyonbothcompressed(i.e.,approximated)andrawtimeseriesdata.
We also show that, in some datasets, wavelets may actually help in reducing noisy
variationswhichdeterioratetheperformanceofTSCtasks.Inafewcases,wenote
that the residual details/noises from compression are more useful for recognizing
data patterns.
In the second part, we propose a language model-based approach for TSC
named Domain Series Corpus (DSCo), in order to take advantage of mature tech-
niquesfrombothtimeseriesminingandNaturalLanguageProcessing(NLP)com-
munities. After transforming real-valued time series into texts using Symbolic Ag-
gregate approXimation(SAX), webuild per-classlanguage models(unigrams and
bigrams) from these symbolized text corpora. To classify unlabeled samples, we
compute the fitness of each symbolized sample against all per-class models and
choose the class represented by the model with the best fitness score. Through
extensive experiments on an open dataset archive, we demonstrate that DSCo per-
forms similarly to approaches working with original uncompressed numeric data.
We further propose DSCo-NG to improve the computation efficiency and classifi-
i
cation accuracy of DSCo. In contrast to DSCo where we try to find the best way
to recursively segment time series, DSCo-NG breaks time series into smaller seg-
mentsofthesamesize,thissimplificationalsoleadstosimplifiedlanguagemodel
inference in the training phase and slightly higher classification accuracy.
Thethirdpartofthisdissertationpresentsamultiscalevisibilitygraphrepresen-
tation for time series as well as feature extraction methods for TSC, so that both
global and local features are fully extracted from time series data. Unlike tradi-
tional TSC approaches that seek to find global similarities in time series databases
(e.g., 1NN-DTW) or methods specializing in locating local patterns/subsequences
(e.g.,shapelets),weextractsolelystatisticalfeaturesfromgraphsthataregenerated
fromtimeseries.Specifically,weaugmenttimeseriesbymeansoftheirmultiscale
approximations, which are further transformed into a set of visibility graphs. Af-
ter extracting probability distributions of small motifs, density, assortativity, etc.,
these features are used for building highly accurate classification models using
generic classifiers (e.g., Support Vector Machine and eXtreme Gradient Boosting).
Based on extensive experiments on a large number of open datasets and compar-
ison with five state-of-the-art TSC algorithms, our approach is shown to be both
accurateandefficient:itismoreaccuratethanLearningShapeletsandatthesame
time faster than Fast Shapelets.
Finally, we list a few industrial applications that relevant to our research work,
including Non-Intrusive Load Monitoring as well as anomaly detection and visu-
alization by means for hierarchical clustering for time series data.
In summary, this dissertation explores different possibilities to improve the effi-
ciency and accuracy of TSC algorithms. To that end, we employ a range of tech-
niques including wavelet transforms, symbolic approximations, language models
and graph mining algorithms. We experiment and evaluate our approaches using
publiclyavailabletimeseriesdatasets.Comparisonwiththestate-of-the-artshows
that the approaches developed in this dissertation perform well, and contribute to
advance the field of TSC.
ii
To my family.

ACKNOWLEDGMENTS
First and foremost, I would like to thank my dissertation advisor Prof. Dr. Yves
LeTraonforhiscontinuoussupportduringmydoctoralstudy.Hehasofferednot
only full support but also the maximum freedom, so that I can conduct research
and explorations in topics I am interested in.
My sincere gratitude also goes to my daily advisors Dr. Jacques Klein and Dr.
Tegawendé F.Bissyandé fortheir guidance andencouragement. Theirsupport has
saved me from having an abysmal time and made the completion of this disserta-
tion all possible (instead of dropping out in the middle).
I am grateful for the jury members for their interest in this dissertation and
taking their valuable time to evaluate this dissertation. In addition, many people
have provided insightful comments for my Ph.D. work. Especially, I would like to
acknowledgeProf.Dr.KarlAbererfromEPFL,Prof.Dr.EamonnKeoghfromUCR
and Prof. Dr. Jessica Lin from GMU for their helpful suggestions on my previous
work on which this dissertation is built.
My special thanks go to Dr. Anne-Marie Solvi, Paul Schummer and colleagues
from Paul Wurth Geprolux S.A.. I have had a wonderful time working on the
smart buildings and PWBox project with them.
Finally, I could not have gone so far (or even started in the first place) without
mybelovedwifeandmyparentsinmyback.Theyarethesunshineinmylifeand
their support has always been warm and unreserved. I cannot imagine surviving
this journey without their support.
Daoyuan Li
Luxembourg
11 2018
January ,
v

CONTENTS
acronyms
xi
list of algorithms
xiii
list of tables
xv
list of figures
xix
overview 1
I
1 introduction 3
11 3
. Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
12 5
. Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
13 6
. Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
14 8
. Organization of this Dissertation . . . . . . . . . . . . . . . . . . . . .
time series 9
II
2 background 11
21 11
. Time Series . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
22 12
. Distance Measures . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
23 15
. Symbolic Representation of Time Series . . . . . . . . . . . . . . . . .
3 state of the art 19
- - -
31 19
. Generic Classification Algorithms . . . . . . . . . . . . . . . . . . . .
3.1.1 kNN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
312 20
. . Support Vector Machine . . . . . . . . . . . . . . . . . . . . . .
313 20
. . Decision Trees . . . . . . . . . . . . . . . . . . . . . . . . . . . .
314 21
. . Ensemble Methods . . . . . . . . . . . . . . . . . . . . . . . . .
315 22
. . Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . . .
32 22
. Time Series Classifiers . . . . . . . . . . . . . . . . . . . . . . . . . . .
321 23
. . Similarity-based Nearest Neighbor . . . . . . . . . . . . . . . .
322 23
. . Bag-of-Patterns . . . . . . . . . . . . . . . . . . . . . . . . . . .
323 24
. . SAX-VSM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
324 24
. . Representative Pattern Mining . . . . . . . . . . . . . . . . . .
325 25
. . BOSS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
326 25
. . Shapelets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
327 26
. . Logical Shapelets . . . . . . . . . . . . . . . . . . . . . . . . . .
328 26
. . Learning Shapelets . . . . . . . . . . . . . . . . . . . . . . . . .
329 26
. . Shapelet Transform . . . . . . . . . . . . . . . . . . . . . . . . .
3210 26
. . Fast Shapelets . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3211 27
. . Collective of Transformation-based Ensembles . . . . . . . . .
33 27
. Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
34 28
. Parameters and Hyper-parameters . . . . . . . . . . . . . . . . . . . .
341 30
. . Cross Validation . . . . . . . . . . . . . . . . . . . . . . . . . .
342 30
. . Tuning Hyper-Parameters . . . . . . . . . . . . . . . . . . . . .
vii
viii contents
transforming time series for tsc 33
III
4 discrete wavelet transform for dimensionality reduction 35
41 35
. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
42 37
. Discrete Wavelet Transform . . . . . . . . . . . . . . . . . . . . . . . .
43 39
. Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
44 40
. Experimental Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
441 40
. . Setup and Datasets . . . . . . . . . . . . . . . . . . . . . . . . .
442 41
. . TSC with Wavelet Transformed Data . . . . . . . . . . . . . .
443 43
. . TSC with Residual Details . . . . . . . . . . . . . . . . . . . . .
444 45
. . Multi-Level Wavelet Transformation . . . . . . . . . . . . . . .
445 47
. . Using the UCR suite for TSC and Significance Test . . . . . .
45 47
. The Smoothing Effect of Wavelets. . . . . . . . . . . . . . . . . . . . .
46 50
. Conclusions and Future Work . . . . . . . . . . . . . . . . . . . . . . .
5 domain series corpora 53
51 53
. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
52 55
. Background and Key Intuition . . . . . . . . . . . . . . . . . . . . . .
521 55
. . Language Modeling . . . . . . . . . . . . . . . . . . . . . . . .
53 56
. Domain Series Corpora for TSC . . . . . . . . . . . . . . . . . . . . . .
531 56
. . Data Representation as Texts . . . . . . . . . . . . . . . . . . .
532 56
. . Language Model Inference . . . . . . . . . . . . . . . . . . . .
533 58
. . Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . .
54 59
. Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
541 60
. . Reducing Data using SAX . . . . . . . . . . . . . . . . . . . . .
542 61
. . Implementation and Setup . . . . . . . . . . . . . . . . . . . .
543 62
. . Comparison of Classification Performance . . . . . . . . . . .
544 65
. . Time and Space Complexity. . . . . . . . . . . . . . . . . . . .
545 66
. . Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
55 66
. Improving DSCo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
551 67
. . Compressing Time Series into Texts . . . . . . . . . . . . . . .
552 67
. . Extracting Language Models . . . . . . . . . . . . . . . . . . .
553 68
. . Classifying Unlabeled Instances . . . . . . . . . . . . . . . . .
554 68
. . Time and Space Complexity. . . . . . . . . . . . . . . . . . . .
56 69
. Experimental Evaluation of DSCo-NG . . . . . . . . . . . . . . . . . .
561 69
. . Implementation and Setup . . . . . . . . . . . . . . . . . . . .
562 69
. . Parameter Optimization . . . . . . . . . . . . . . . . . . . . . .
563 70
. . Comparison of Classification Performance . . . . . . . . . . .
57 73
. Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
58 75
. Conclusions and Future Work . . . . . . . . . . . . . . . . . . . . . . .
6 multiscale visibility graph 77
61 77
. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
62 80
. Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
621 81
. . Visibility Graph . . . . . . . . . . . . . . . . . . . . . . . . . . .
622 83
. . Graph Classification with Deep Neural Networks . . . . . . .
623 84
. . Graph Features . . . . . . . . . . . . . . . . . . . . . . . . . . .
63 88
. Multiscale Visibility Graph . . . . . . . . . . . . . . . . . . . . . . . .
631 90
. . Feature Extraction . . . . . . . . . . . . . . . . . . . . . . . . .