Table Of ContentAdvanced Information and Knowledge Processing
Series Editors
Professor Lakhmi Jain
[email protected]
Professor Xindong Wu
[email protected]
Also in this series
Gregoris Mentzas, Dimitris Apostolou, Andreas Abecker and Ron Young
Knowledge Asset Management
1-85233-583-1
Michalis Vazirgiannis, Maria Halkidi and Dimitrios Gunopulos
Uncertainty Handling and Quality Assessment in Data Mining
1-85233-655-2
Asunción Gómez-Pérez, Mariano Fernández-López and Oscar Corcho
Ontological Engineering
1-85233-551-3
Arno Scharl (Ed.)
Environmental Online Communication
1-85233-783-4
Shichao Zhang, Chengqi Zhang and Xindong Wu
Knowledge Discovery in Multiple Databases
1-85233-703-6
Jason T.L. Wang, Mohammed J. Zaki, Hannu T.T. Toivonen and Dennis Shasha (Eds)
Data Mining in Bioinformatics
1-85233-671-4
C.C. Ko, Ben M. Chen and Jianping Chen
Creating Web-based Laboratories
1-85233-837-7
Manuel Graña, Richard Duro, Alicia d’Anjou and Paul P. Wang (Eds)
Information Processing with Evolutionary Algorithms
1-85233-886-0
Colin Fyfe
Hebbian Learning and Negative Feedback Networks
1-85233-883-0
Yun-Heh Chen-Burger and Dave Robertson
Automating Business Modelling
1-85233-835-0
Dirk Husmeier, Richard Dybowski and Stephen Roberts (Eds)
Probabilistic Modeling in Bioinformatics and Medical Informatics
1-85233-778-8
Ajith Abraham, Lakhmi Jain and Robert Goldberg (Eds)
Evolutionary Multiobjective Optimization
1-85233-787-7
K.C. Tan, E.F. Khor and T.H. Lee
Multiobjective Evolutionary Algorithms and Applications
1-85233-836-9
Nikhil R. Pal and Lakhmi Jain (Eds)
Advanced Techniques in Knowledge Discovery and Data Mining
1-85233-867-9
Amit Konar and Lakhmi Jain
Cognitive Engineering
1-85233-975-6
Miroslav Kárny´ (Ed.)
Optimized Bayesian Dynamic Advising
1-85233-928-4
Marcus A. Maloof (Ed.)
Machine Learning and Data Mining for Computer Security
1-84628-029-X
Yannis Manolopoulos, Alexandros Nanopoulos, Apostolos N. Papadopoulos and
Yannis Theodoridis
R-trees: Theory and Applications
1-85233-977-2
Sanghamitra Bandyopadhyay, Ujjwal Maulik, Lawrence B. Holder and Diane J. Cook (Eds)
Advanced Methods for Knowledge Discovery from Complex Data
1-85233-989-6
Sifeng Liu and Yi Lin
Grey Information
1-85233-995-0
Mitra Basu and Tin Kam Ho (Eds)
Data Complexity in
Pattern Recognition
Mitra Basu,PhD Tim Kam Ho,BBA,MS,PhD
Electrical Engineering Department Bell Laboratories
City College Lucent Technologies
City University of New York New Jersey
USA USA
British Library Cataloguing in Publication Data
A catalogue record for this book is available from the British Library
Library ofCongress Control Number:2005936159
Advanced Information and Knowledge Processing ISSN 1610-394
ISBN-10:1-84628-171-7 e-ISBN 1-84628-172-5
ISBN-13:978-1-84628-171-6
Printed on acid-free paper
© Springer-Verlag London Limited 2006
Apart from any fair dealing for the purposes ofresearch or private study,or criticism or review,
as permitted under the Copyright,Designs and Patents Act 1988,this publication may only be
reproduced,stored or transmitted,in any form or by any means,with the prior permission in
writing of the publishers,or in the case of reprographic reproduction in accordance with the
terms oflicences issued by the Copyright Licensing Agency.Enquiries concerning reproduction
outside those terms should be sent to the publishers.
The use of registered names, trademarks, etc. in this publication does not imply, even in
the absence of a specific statement, that such names are exempt from the relevant laws and
regulations and therefore free for general use.
The publisher makes no representation,express or implied,with regard to the accuracy of the
information contained in this book and cannot accept any legal responsibility or liability for any
errors or omissions that may be made.
Printed in the United States ofAmerica (MVY)
9 8 7 6 5 4 3 2 1
springer.com
Preface
Machines capable of automatic pattern recognition have been fascinating subjects
that capture much popular imagination. Algorithms for supervised classification,
whereoneinfersadecisionboundaryfromasetoftrainingexamples,areatthecore
of this capability. Over the last few decades, tremendous progress has been made
on inventing and refining such algorithms. Yet, automatic learning in many simple
tasksindailylifestillappearstobefarfromreach.Soweask,whatismissingfrom
theseefforts?Whentheautomaticclassifiersarenotperfect,isitadeficiencyofthe
algorithmsbydesign,orisitadifficultyintrinsictotheclassificationtask?Howdo
we know whether we have exploited to the fullest extent the knowledge embedded
inthetrainingdata?
A well-known description of the intrinsic difficulty of a classification problem
is the Bayes error, which can be computed if complete knowledge is given on the
probabilitydistributionofeachclassinthefeaturespace.Butempiricalobservations
suggest that, from the perspectives of automatic classifiers given limited training
data,problemscanbedifficultfordifferentreasonseveniftheyhavethesameBayes
error.Thisresultsinstronglydata-dependentperformancesofclassifiers.Factorsaf-
fecting accuracy can be, for example, the shapes of the classes and thus the shape
the decision boundary, the amount of overlap among the classes, the proximity of
twoclasses,andthenumberofinformativesamplesavailablefortraining.Classifiers
respond to these factors in their unique ways. A recent realization is that the com-
plexityofthedatamustbeweighedinwhenevaluatingtheperformanceofvarious
classifiers.
Besides pattern recognition, classification is also a central topic in several re-
lateddisciplinessuchasmachinelearning,neuralnetworks,anddatamining.Prac-
ticalapplicationsarealsonumerous,inimage,video,andspeechsignalprocessing,
biometrics,digitallibraries,biomedicalstudies,financialengineering,militarydeci-
sions,andmanyotherareasofscientific,engineering,andbusinessdataanalysis.A
prerequisiteforsettingproperexpectationsonclassificationperformanceistounder-
standthecomplexityofaspecificdatasetarisingfromanapplication.Tounderstand
the data complexity is to find out whether, or to what extent, patterns exist in the
VI Preface
data.Itisalsotoobtainguidanceonselectingspecificclassificationtechniques.We
believethatthisisthekeytofurtheradvancesinclassification.
Wehaveseenrelevantattemptstounderstanddatacharacteristicsinseveralma-
jor methodological areas. However, the huge and diverse literature of the field has
made it extremely difficult for one to keep up with the progress. In addition, be-
causeofbackgroundandterminologydifferences,ithasbecomeasadstateofaffairs
thatmanyresearchersareoftenunawareofconcurrentdevelopmentsunderslightly
differentnames.Motivatedbythis,wecollectedinthisbookasetofreviews,com-
mentaries, and case studies on data complexity and its role in shaping the theories
andtechniquesindifferentareas.
The book consists of two parts: (I) Theory and Methodology, and (II) Applica-
tions.Part Ibegins withseveral chapters thatproposemeasures ofdatacomplexity
fromdifferentperspectives.Chapter1describesseveralmeasuresofgeometricaland
topologicalcharacteristicsofpointsetsinhigh-dimensionalspaces,andtheirutility
in analyzing data sets with known or controlled complexity. Chapter 2 focuses on
a dissimilarity based representation of objects and addresses the intricate relation-
shipbetweentheintrinsiccomplexityofadataset,thechosenobjectrepresentation,
the apparent complexity determined by the sampling density with respect to this
representation, along with the implications of these on classifier choices. Chapter
3discussesthepossibilitiesanddifficultiesinestimatingdatacomplexitywhenthe
datamodelsareassumedtobeinaknownform,suchasanisotropicGaussian,for
which certain simple classifiers are suitable. Examples are shown where the same
problemmayappeareasyordifficultdependingontheclassifierbeingused,dueto
influences of the density parameters and the sample size on the relevant measures.
The chapter concludes with an interesting remark that estimating such measures is
notnecessarilyeasierthantrainingaclassifierforthetask.
Various aspects of data complexity can affect the behavior of different families
of classifiers in different ways. Among the earliest concerns is the role of linear
separability on the effectiveness of linear classifiers. This concern is carried into
recentstudiesonsupportvectormachinesthatarelinearclassifiersinatransformed
featurespace.Chapter4reviewsdescentproceduresforlinearclassifiertraining,and
discussestheeffectoflinearseparabilityonthebehaviorofsuchprocedures.Chapter
5relatesdatacomplexitytotheinductivelearningprincipleandthecontrolofmodel
complexity.Twoapproachestocontrolclassifiercomplexityarecompared:amargin-
basedapproachandanadaptiveparameterizationapproach.Experimentsareshown
thathighlightsconditionsunderwhichthemargin-basedapproachismoreeffective.
Chapter6describestheinfluenceofdatageometryontheperformanceofapop-
ular genetic algorithm based classifier, XCS. The chapter continues to use several
geometrical complexity measures to analyze the regions in the complexity space
whereXCSperformsfavorably.Thisstudyisextendedtosixclassifiersinchapter7,
wherethedomainsofdominantcompetencearefoundforthreetraditionalclassifiers
andthreetypesofensembleclassifiers.
Grammatical inference has received much attention recently. Concerted efforts
from diverse disciplines to find tractable inference techniques have added new di-
mensionsandopenedupunexploredterritories.Chapter8studiesthedatacomplex-
Preface VII
ity issue in grammatical inference at three concentric levels: representational level
(strings,trees,andgraphs),classoflanguages,andtheissueofmembership.Choices
madeatonelevelmayaffectdatacomplexityasseenatanotherlevel.Forexample,
thechoiceintherepresentationallevelmaymakethelatertaskseasierorharder.The
authorreviewsnecessarybasicconceptsandpresentsdiscussionsonhowstochastic
andotherlanguageshaveevolvedtoaddresstheneeds.
The question of data complexity affects all applications of classification, in re-
vealingorrejectingexistenceofdependences,suggestingandjustifyingaparticular
classificationmethod,anddeterminingtheutilityofpredictions.Forexample,stud-
ies on data complexity can help explore the existence of patterns in occurrences
of diseases, characteristics of dependencies in retail transactions, reliability of bio-
metrical authentication techniques, feasibility of earthquake prediction by seismic
waveforms,andthepracticalityofcategorizingregulartext,emailmessage,andweb
pages.Similarproblemsalsooccurinmanyotherareasofscienceandengineering.
In Part II, we collected studies of several key applications on their sources of
difficultiesandhowdatacomplexityinfluencesthecomplexityandbehavioroftheir
solutions.
Chapter 9 features a large set of detailed analyses of properties of pattern dis-
tributionsinahigh-dimensionalfeaturespaceassociatedwithasymbolrecognition
problem,bothfromtheoryandfromempiricalobservations.Thetoolsandarguments
developed there can be useful to many other application areas involving natural or
artificialpatterns.
Chapter 10 proposes a set of graph-representation–based methods suitable for
document comparison, with special motivations for usage in a web mining con-
text. The methods feature compact representation, practical speed, and comparable
or even favorable accuracy over traditional vector-based methods. This exemplifies
howmethodsaredesignedtoadapttothedifficultyofaproblem.
Chapter 11 reviews clustering algorithms that are applied to microarray data,
showingthattheclassicalalgorithmsareinadequatetoextracttheunderlyingchar-
acteristics.Theauthorscallattentiontoissuesfacingthebioinformaticscommunity
anddescribesattemptsinthiscommunitytoredesignclusteringalgorithmsthatare
sensitivetodatacomplexityinbiologicaldata.
Chapter12presentstheproblemofclassifyingmagneticresonancespectrawhere
high dimensionality of the raw feature space and extreme sample sparsity have
caused many difficulties. Classification accuracies are strongly influenced by the
problem representation using different extracted features. Geometrical complexity
offersanexplanationforthisinfluence.
Chapter13givesatutorialintroductiontotheproblemoftropicalcyclonetrack-
ing in weather forecasting, and discusses the various aspects of the problem where
complexitiesinthedatahavestrongimpactsonthepredictionaccuracy.
Chapter 14 argues strongly for involving humans in the loop of developing a
solution to a visual pattern recognition problem where automatic methods may not
beabletofullyadapttodatacomplexity.ThechaterdescribestheCAVIARsoftware
thatservesasaproofoftheconcept.
VIII Preface
Finally, is classification complexity always something to be avoided? It turns
out that there are occasions where one actually wishes to exploit the difficulty of
a machine vision problem for some good use. Chapter 15 describes a new topic of
studycalled“humaninteractiveproof,”wherethesecurityofaninformationsystem
is guarded by “reverse Turing tests”that present patterns difficult for machines but
easyforhumantorecognize.
Datacomplexityisafundamental,important,andfar-reachingtopic.Ourattempt
in this book captures only a snapshot of the recent activities related to this theme.
Therehavebeenmanyotherrelatedinvestigationsinstatisticsandinformationthe-
ory. Although it is not possible to survey all the relevant past and current works in
thissinglevolume,wehopethiscollectionwillbringattentiontothistopic,andthat
thereferencesincludedherewillprovidesomebackgroundtoresearchersinterested
in following up this development in pattern recognition. It is our belief that mak-
ingthesediscussionsaccessibletoawideraudiencewillstimulatefurtherexchanges
amongvariouscommunitiesdealingwithdataanalysis.
We are grateful to all the contributors and reviewers who have made this en-
deavoranenjoyableandexcitinglearningexperience.Theeditorialandproduction
staffatSpringerhaveprovidedvaluablesupportduringthepreparationofthemanu-
script.WearealsoindebttoAnilJain,RobertDuin,andGeorgeNagy,whokindly
providedmuchencouragementintheearlyphasesofthisstudy.Tinthanksthesup-
portofBellLabsmanagementonherpursuits,andinparticular,LawrenceCowsar,
WimSweldens,AlAho,MargaretWright,andBillCoughran,forhostingvisitsfrom
Mitra,George,Ester,andMartinoverthelastfewyearswhilethisworkdeveloped.
MitraBasu,Arlington,VA
TinKamHo,MurrayHill,NJ
Contents
ListofContributors ................................................ XI
PartI TheoryandMethodology
1 MeasuresofGeometricalComplexityinClassificationProblems
TinKamHo,MitraBasu,andMartinHiuChungLaw ................... 3
2 ObjectRepresentation,SampleSize,andDataSetComplexity
RobertP.W.DuinandElz˙bietaPe.kalska ............................... 25
3 MeasuresofDataandClassifierComplexityandtheTrainingSample
Size
Sˇaru¯nasRaudys................................................... 59
4 LinearSeparabilityinDescentProceduresforLinearClassifiers
MitraBasuandTinKamHo......................................... 69
5 DataComplexity,Margin-BasedLearning,andPopper’sPhilosophy
ofInductiveLearning
VladimirCherkasskyandYunqianMa ................................. 91
6 DataComplexityandEvolutionaryLearning
EsterBernado´-Mansilla,TinKamHo,andAlbertOrriols .................115
7 ClassifierDomainsofCompetenceinDataComplexitySpace
TinKamHoandEsterBernado´-Mansilla ..............................135
8 DataComplexityIssuesinGrammaticalInference
ColindelaHiguera ...............................................153
X Contents
PartII Applications
9 SimpleStatisticsforComplexFeatureSpaces
GeorgeNagyandXiaoliZhang ......................................173
10 PolynomialTimeComplexityGraphDistanceComputationforWeb
ContentMining
AdamSchenker,HorstBunke,MarkLast,andAbrahamKandel.............197
11 Data Complexity in Clustering Analysis of Gene Microarray
ExpressionProfiles
FengLuoandLatifurKhan..........................................217
12 ComplexityofMagneticResonanceSpectrumClassification
RichardBaumgartner,TinKamHo,RaySomorjai,UweHimmelreich,Tania
Sorrell ..........................................................241
13 DataComplexityinTropicalCyclonePositioningandClassification
ChiLapYip,KaYanWong,andPingWahLi............................249
14 Human–ComputerInteractionforComplexPatternRecognition
Problems
JieZouandGeorgeNagy ...........................................271
15 ComplexImageRecognitionandWebSecurity
HenryS.Baird....................................................287
Index ............................................................. 299
