Unsupervised Learning Algorithms

564 Pages·2016·13.96 MB·English

by  M. Emre Celebi

M. Emre Celebi · Kemal Aydin Editors
Unsupervised
Learning
Algorithms
Unsupervised Learning Algorithms

M. Emre Celebi • Kemal Aydin
Editors
Unsupervised Learning
Algorithms
123
Editors
M.EmreCelebi KemalAydin
DepartmentofComputerScience NorthAmericanUniversity
LouisianaStateUniversityinShreveport Houston,TX,USA
Shreveport,LA,USA
ISBN978-3-319-24209-5 ISBN978-3-319-24211-8 (eBook)
DOI10.1007/978-3-319-24211-8
LibraryofCongressControlNumber:2015060229
SpringerChamHeidelbergNewYorkDordrechtLondon
©SpringerInternationalPublishingSwitzerland2016
Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpartof
thematerialisconcerned,specificallytherightsoftranslation,reprinting,reuseofillustrations,recitation,
broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,andtransmissionorinformation
storageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilarmethodology
nowknownorhereafterdeveloped.
Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication
doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant
protectivelawsandregulationsandthereforefreeforgeneraluse.
Thepublisher,theauthorsandtheeditorsaresafetoassumethattheadviceandinformationinthisbook
arebelievedtobetrueandaccurateatthedateofpublication.Neitherthepublishernortheauthorsor
theeditorsgiveawarranty,expressorimplied,withrespecttothematerialcontainedhereinorforany
errorsoromissionsthatmayhavebeenmade.
Printedonacid-freepaper
SpringerInternational PublishingAGSwitzerlandispartofSpringerScience+Business Media(www.
springer.com)
Preface
Withtheproliferationofmassiveamountsofunlabeleddata,unsupervisedlearning
algorithms–whichcanautomaticallydiscoverinterestingandusefulpatternsinsuch
data–havegainedpopularityamongresearchersandpractitioners.Thesealgorithms
have found numerous applications including pattern recognition, market basket
analysis,webmining,socialnetworkanalysis,informationretrieval,recommender
systems,marketresearch,intrusiondetection,andfrauddetection.Thedifficultyof
developingtheoreticallysoundapproachesthatareamenabletoobjectiveevaluation
hasresultedintheproposalofnumerousunsupervisedlearningalgorithmsoverthe
pasthalf-century.
The goal of this volume is to summarize the state of the art in unsupervised
learning. The intended audience includes researchers and practitioners who are
increasinglyusingunsupervisedlearningalgorithmstoanalyzetheirdata.
This volume opens with two chapters on anomaly detection. In “Anomaly
Detection for Data with Spatial Attributes,” P. Deepak reviews anomaly detection
techniquesforspatialdatadevelopedinthedataminingandstatisticscommunities.
The author presents a taxonomy of such techniques, describes the most represen-
tativeones,anddiscussestheapplicationsofclusteringandimagesegmentationto
anomalydetection.
In“AnomalyRankinginaHighDimensionalSpace:TheUnsupervisedTreeR-
ank Algorithm,” Clémençon et al. describe a computationally efficient anomaly
ranking algorithm based on the minimization of the mass-volume criterion. This
algorithmdoesnotinvolveanysampling;therefore,itisespeciallysuitedforlarge
andhigh-dimensionaldata.
Clustering isundoubtedlythe mostwell-knownsubfieldof unsupervisedlearn-
ing.Thevolumecontinueswith12chaptersonclustering.In“GeneticAlgorithms
for Subset Selection in Model-Based Clustering,” Scrucca describes a genetic
algorithmthatmaximizestheBayesianinformationcriterion(BIC)toselectasubset
ofrelevantfeaturesformodel-basedclustering.Inparticular,thecriterionisbased
on the BIC difference between a candidate clustering model for the given subset
andamodel,whichassumesnoclusteringforthesamesubset.Theimplementation
v
vi Preface
ofthisalgorithmusesthefacilitiesavailableintheGApackagefortheopen-source
statisticalcomputingenvironmentR.
In “Clustering Evaluation in High-Dimensional Data,” Tomašev and
Radovanovic´ investigate the performance of popular cluster quality assessment
indices on synthetically generated high-dimensional Gaussian data. Extensive
experiments reveal that dimensionality and degree of cluster overlap can affect
both the mean quality score assigned by an index and the stability of its quality
estimation.Theauthorsalsodiscoverthatappropriatetreatmentofhubpointsmay
improvethequalityassessmentprocess.
In“CombinatorialOptimizationApproachesforDataClustering,”Festapresents
anoverviewofclusteringalgorithmswithparticularemphasisonalgorithmsbased
on combinatorial optimization. The author first reviews various mathematical
programming formulations of the partitional clustering problem and some exact
methodsbased on them. She then providesa brief surveyof partitionalclustering
algorithmsbasedonheuristicsormetaheuristics.
In“KernelSpectralClusteringandApplications,”Langoneetal.presentasurvey
of the recently proposed kernel spectral clustering (KSC) algorithm. The authors
describe the basic KSC algorithm as well as its probabilistic, hierarchical, and
sparseextensions.Theyalsoprovideanoverviewofthevariousapplicationsofthe
algorithmsuch as text clustering, image segmentation,power load clustering, and
communitydetectioninbigdatanetworks.
In“Uni-andMulti-DimensionalClusteringviaBayesianNetworks,”Keivaniand
Peñadiscussmodel-basedclusteringusingBayesiannetworks.Forunidimensional
clustering,theauthorsproposetheuseoftheBayesianstructuralclustering(BSC)
algorithm, which is based on the celebrated expectation-maximization algorithm.
For the multidimensionalcase, the authors propose two algorithms, one based on
a generalization of the BSC algorithm and the other based on multidimensional
Bayesiannetworkclassification.Theformeralgorithmturnsouttobecomputation-
allydemanding.So,theauthorsprovideapreliminaryevaluationofthelatterontwo
representativedatasets.
In “A Radial Basis Function Neural Network Training Mechanism for Pattern
ClassificationTasks,”NirosandTsekourasproposeanovelapproachfordesigning
radialbasisfunctionnetworks(RBFNs)basedonhierarchicalfuzzyclusteringand
particle swarm optimization with discriminant analysis. The authors compare the
resultingRBFNclassifieragainstvariousotherclassifiersonpopulardatasetsfrom
theUCImachinelearningrepository.
In “A Surveyof ConstrainedClustering,” Dinler and Tural providean in-depth
overviewof the field of constrainedclustering (a.k.a. semi-supervised clustering).
After giving an introduction to the field of clustering, the authors first review
unsupervisedclustering.Theythenpresentasurveyofconstrainedclustering,where
the prior knowledge comes from either labeled data or constraints. Finally, they
discusscomputationalcomplexityissuesandrelatedworkinthefield.
In “An Overview of the Use of Clustering for Data Privacy,” Torra et al. give
abriefoverviewofdataprivacywithemphasisontheapplicationsofclusteringto
Preface vii
data-drivenmethods.Morespecifically,theyreviewtheuseofclusteringinmasking
methodsandinformationlossmeasures.
In “Nonlinear Clustering: Methods and Applications,” Wang and Lai review
clustering algorithms for nonlinearly separable data. The authors focus on four
approaches:kernel-basedclustering,multi-exemplar-basedclustering,graph-based
clustering, and support vector clustering. In addition to discussing representative
algorithmsbased on each of these approaches,the authorspresentapplicationsof
these algorithms to computer vision tasks such as image/video segmentation and
imagecategorization.
In “Swarm Intelligence-Based Clustering Algorithms: A Survey,”˙Inkaya et al.
presenta detailedsurveyofalgorithmsforhardclusteringbasedonswarmintelli-
gence(SI).TheycategorizeSIbasedclusteringalgorithmsintofivegroups:particle
swarm optimization based algorithms, ant colony optimization based algorithms,
ant based sorting algorithms,hybridalgorithms, and miscellaneousalgorithms.In
addition,theypresentanoveltaxonomyforSIbasedclusteringalgorithmsbasedon
agentrepresentation.
In “ExtendingKmeans-TypeAlgorithmsbyIntegratingIntra-ClusterCompact-
nessandInter-ClusterSeparation,”Huangetal.proposeaframeworkforintegrating
bothintra-clustercompactnessandinter-clusterseparationcriteriaink-means-type
clusteringalgorithms.Basedontheirproposedframework,theauthorsdesignthree
novel,computationallyefficientk-means-typealgorithms.Theperformanceofthese
algorithms is demonstrated on a variety of data sets, using several cluster quality
assessmentindices.
In “A Fuzzy-Soft Competitive Learning Approach for Grayscale Image Com-
pression,” Tsolakis and Tsekouras propose a novel, two-stage vector quantization
algorithmthatcombinesthemeritsofhardandsoftvectorquantizationparadigms.
Thefirststageinvolvesa softcompetitivelearningschemewitha fuzzyneighbor-
hoodfunction,which can measure the lateral neuroninteraction phenomenonand
thedegreeofneuronexcitations.Thesecondstageimprovesthepartitiongenerated
in the first stage by meansof a codewordmigrationstrategy.Experimentalresults
on classic grayscale images demonstrate the effectiveness and efficiency of the
proposedalgorithmincomparisonwithseveralstate-of-the-artalgorithms.
This volume continues with two chapters on the applications of unsupervised
learning.In“UnsupervisedLearninginGenomeInformatics,”Wongetal.reviewa
selectionofstate-of-the-artunsupervisedlearningalgorithmsforgenomeinformat-
ics.Thechapterisdividedintotwoparts.Inthefirstpart,theauthorsreviewvarious
algorithms for protein-DNA binding event discovery and search from sequence
patternstogenome-widelevels.Inthesecondpart,severalalgorithmsforinferring
microRNAregulatorynetworksarepresented.
In “The Application of LSA to the Evaluation of Questionnaire Responses,”
Martinetal.investigatetheapplicabilityofLatentSemanticAnalysis(LSA)tothe
automatedevaluationofresponsestoessayquestions.Theauthorsfirstdescribethe
natureofessayquestions.TheythengiveadetailedoverviewofLSAincludingits
historicalandmathematicalbackground,itsuseasanunsupervisedlearningsystem,
viii Preface
and its applications. The authors conclude with a discussion of the application of
LSA to automated essay evaluation and a case study involving a driver training
system.
This volume concludes with two chapters on miscellaneous topics regarding
unsupervised learning. In “Mining Evolving Patterns in Dynamic Relational Net-
works,”AhmedandKarypispresentvariouspracticalalgorithmsforunsupervised
analysis of the temporal evolution of patterns in dynamic relational networks.
The authors introduce various classes of dynamic patterns, which enable the
identificationofhiddencoordinationmechanismsunderlyingthenetworks,provide
information on the recurrence and stability of its relational patterns, and improve
theabilitytopredicttherelationsandtheirchangesinthesenetworks.
Finally, in “Probabilistically Grounded Unsupervised Training of Neural Net-
works,” Trentin and Bongini present a survey of probabilistic interpretations of
artificial neural networks (ANNs). The authors first review the use of ANNs
for estimating probability density functions. They then describe a competitive
neuralnetworkalgorithmforunsupervisedclusteringbasedonmaximumlikelihood
estimation.Theyconcludewithadiscussionofprobabilisticmodelingofsequences
ofrandomobservationsusingahybridANN/hiddenMarkovmodel.
We hope that this volume, focused on unsupervised learning algorithms, will
demonstratethe significantprogressthathas occurredin this field in recentyears.
We also hope that the developmentsreported in this volume will motivate further
researchinthisexcitingfield.
Shreveport,LA,USA M.EmreCelebi
Houston,TX,USA KemalAydin
Contents
AnomalyDetectionforDatawithSpatialAttributes......................... 1
P.Deepak
Anomaly Ranking in a High Dimensional Space:
TheUNSUPERVISED TREERANKAlgorithm ................................. 33
S.Clémençon,N.Baskiotis,andN.Vayatis
GeneticAlgorithmsforSubsetSelectioninModel-BasedClustering...... 55
LucaScrucca
ClusteringEvaluationinHigh-DimensionalData............................ 71
NenadTomaševandMilošRadovanovic´
CombinatorialOptimizationApproachesforDataClustering ............. 109
PaolaFesta
KernelSpectralClusteringandApplications................................. 135
Rocco Langone, Raghvendra Mall, Carlos Alzate,
andJohanA.K.Suykens
Uni-andMulti-DimensionalClusteringViaBayesianNetworks........... 163
OmidKeivaniandJoseM.Peña
ARadialBasisFunctionNeuralNetworkTrainingMechanism
forPatternClassificationTasks................................................. 193
AntoniosD.NirosandGeorgeE.Tsekouras
ASurveyofConstrainedClustering........................................... 207
DeryaDinlerandMustafaKemalTural
AnOverviewoftheUseofClusteringforDataPrivacy..................... 237
VicençTorra,GuillermoNavarro-Arribas,andKlaraStokes
NonlinearClustering:MethodsandApplications............................ 253
Chang-DongWangandJian-HuangLai
ix