Efficient Image Set Classification using Linear Regression based Image Reconstruction

0.5 MB·

by  Syed Afaq Ali Shah

Efficient Image Set Classification
using Linear Regression based Image Reconstruction
S.A.A.Shah∗†,U.Nadeem∗†,M.Bennamoun†,F.Sohel‡,andR.Togneri†
†TheUniversityofWesternAustralia
‡MurdochUniversity
7
1
†
0 {afaq.shah@, uzair.nadeem@research., mohammed.bennamoun@, roberto.togneri@} uwa.edu.au
2 ‡[email protected]

n
a
J
Abstract
0
1
We propose a novel image set classification technique
]
V using linear regression models. Downsampled gallery im-
age sets are interpreted as subspaces of a high dimen-
C
sional space to avoid the computationally expensive train-
.
s ingstep. Weestimateregressionmodelsforeachtestimage
c
using the class specific gallery subspaces. Images of the
[
test set are then reconstructed using the regression mod-

1
els. Based on the minimum reconstruction error between
v
thereconstructedandtheoriginalimages, aweightedvot-
5
8 ingstrategyisusedtoclassifythetestset.Weperformedex-
4 tensive evaluation on the benchmark UCSD/Honda, CMU
2 Mobo and YouTube Celebrity datasets for face classifica-
0
tion, and ETH-80 dataset for object classification. The
.
1 results demonstrate that by using only a small amount of
0 training data, our technique achieved competitive classi- Figure1.Ablockdiagramoftheproposedtechnique.
7
fication accuracy and superior computational speed com-
1
paredwiththestate-of-the-artmethods.
: appearance variations within images including: viewpoint
v
i changes, occlusions, non-rigid deformation, variations in
X
illumination and different backgrounds. Because of these
r 1.Introduction characteristics, image set classification has been applied
a
in many applications in biometrics including surveillance,
Image set classification is defined as the problem of
videobasedfacerecognitionandpersonre-identificationin
recognition from multiple images [14]. In image set clas-
anetworkofsecuritycameras[11].
sification,thegalleryortrainingsetconsistsofoneormore
Several image set classification techniques have been
imagesetsforeachclassandeachimage-setcontainsmul-
proposedintheliterature.Afewofthesetechniques,known
tiple images of the same class [14]. The test set also con-
as parametric methods [1], model image sets with certain
tainsanumberofimagesofthesamesubjectwhicharethen
statistical distributions and then calculate the simlarity be-
matched with the training image sets by computing some
tweenthosedistributions.However,thesemethodsrequirea
similarity measure to find the identity of the test subject.
strongstatisticalrelationshipbetweenthetestandthetrain-
Comparedwithtraditionalsingleimagebasedrecognition,
ing image sets to achieve good performance. As opposed
image set classification offers several advantages. For in-
tothesemethods,non-parametricmethodsrepresentimage
stance, image sets can effectively handle a wide variety of
sets as linear or nonlinear subspaces [19], [28], [31], [32].
∗Thefirsttwoauthorscontributedequallytothiswork Thesemethodshaveshownpromisingresultsandarebeing
activelyresearched. The rest of this paper is organized as follows. An
In this paper, we propose a novel non-parametric ap- overviewofrelatedworkispresentedinSection2. Section
proachforimagesetclassification. Theproposedtechnique 3 discusses the proposed technique. Experimental results
isbasedontheconceptofimagereconstructionusingLin- and detailed evaluation of the proposed technique against
ear Regression Classification (LRC) [17] and nearest sub- state-of-the-artapproachesarepresentedinSection4. The
spaceclassification. LRCusestheconceptthatsamplesof comparison of computational time of the proposed tech-
an image category lie on a linear subspace [2], [3]. In our nique with other methods is presented in Section 5. The
proposedtechnique,thegalleryimagesetofeachcategory technique is compared with other latest image set classifi-
forms a subspace in a high dimensional space while using cationmethodsinSection6andconcludedinSection7.
thedownsampledimagesofeachgalleryimageset. Attest
time,eachimageinthetestimagesetisrepresentedasalin- 2.RelatedWork
earcombinationofimagesineachgalleryimageset.Aleast
Imagesetclassificationtechniquescanbecategorizedas
squares based solution is used to estimate the regression
parametric, non-parametric and deep learning based meth-
modelparametersforeachimageofthetestimageset. The
ods. The parametric methods [1] use a statistical distribu-
estimated regression model is used to reconstruct the test
tionmodeltoapproximateanimagesetandthenusesKL-
image from the gallery subspace. The Euclidean distance
divergence to measure the similarity between the two dis-
between the actual test image and the reconstructed image
tribution models. Such methods, however, fail to produce
is then used as the distance metric. Next, weighted voting
good results in case of a weak statistical relationship be-
isusedwhereeachimageofthetestimagesetcastsavote
tweenthetrainingandthetestimagesets.
for each class in the gallery. Finally, the decision rules in
For non-parametric methods, several different metrics
favoroftheclasswiththehighestaccumulatedweight.Fig-
are used to determine the set to set similarity. Wang et
ure 1 shows the block diagram of the proposed technique.
al. [29],usetheEuclideandistancebetweenthesets’mean
Theperformanceoftheproposedtechniquehasbeentested
as the similarity metric. Cevikalp and Triggs [4] present
onfourpopularimagesetclassificationdatasetsCMUMo-
two models to learn set samples. The set to set distance
tion of Body (CMU MoBo) Dataset [8], YoutubeCelebrity
using an affine hull model is called Affine Hull Image Set
(YTC)Dataset[13]andUCSD/HondaDataset[15]forface
Distance (AHISD) while that using convex hull model is
recognitionandETH-80dataset[16]forobjectrecognition.
termed as the Convex Hull Image Set Distance (CHISD).
Weprovidecomparisonwith13imagesetclassificational-
Huetal. [12]usedthemeanimageofimagesetandaffine
gorithms. Themaincontributionsofthispapercanbesum-
hull model to calculate the Sparse Approximated Nearest
marizedasfollows:
Points(SANP)forimagesetsinordertodeterminethedis-
• AnovelextensionofLRCforimagesetclassification, tance between the training image set and test image set.
which is capable of producing state of the art results Some non-parametric methods (e.g., [9], [27], [28], [29])
underthechallengesoflowresolutionandlesstraining useapointonageometricsurfacetorepresentthecomplete
data. Thetechniquedoesnotrequireanytrainingand imageset.Theimagesetcanalsoberepresentedeitherbya
caneasilybegeneralizedacrossdifferentdatasets. combinationoflinearsubspacesoronacomplexnon-linear
manifold. For linear subspaces, the cosine of the smallest
• Since LRC uses least squares solution, any technique
angle between any vector in one subspace and any other
usingLRCispronetotheproblemofsingularmatrix
vectorintheothersubspaceiscommonlyusedasthesimi-
or singularity. This occurs when the rank is less than
laritymetricbetweenimagesets.
thenumberofrowsintheregressor,aconditionknown
Discriminantanalysisiscommonlyusedtorepresentim-
asrankdeficientmatrix. Whilethisproblemismostly
agesetsonthemanifoldsurfacee.g.,DiscriminativeCanon-
ignored in the previous works of image set classifi-
icalCorrelations(DCC)[14],ManifoldDiscriminantAnal-
cation, we present practical and efficient solutions to
ysis (MDA) [27], Graph Embedding Discriminant Analy-
overcometheproblemofarankdeficientmatrix. The
sis (GEDA) [9] and Covariance Discriminative Learning
solutionisnotlimitedtoourtechniqueandcanbegen-
(CDL) [28]. Chen [5] considered training and test image
eralized to any method using LRC and least squares
setsassubspacesinahighdimensionalspaceandusedthe
solutions.
last image of each image set along with the variations of
• The techniques performing operations on each image compared image sets through linear regression classifica-
ofthetestsetareusuallyveryslowandunsuitablefor tion to determine the distance between the two subspaces.
real time applications. On the other hand our tech- Fengetal. [6]extendedtheworkof[5]byusingthemean
niqueusesanefficientmatriximplementationofLRC image of image sets instead of the last image. Moreover
toachievethefastesttesttimecomparedtootherimage theyalsoutilizedtheinformationwhichmaximizesthedis-
setclassificationmethods. tancebetweenthetestsetandunrelatedtrainingsets. How-
ever,forthesemethods,thedimensionofthefeaturevectors vectors xm,m = 1,2,3,...,M are concatenated horizon-
µ
shouldbemuchlargerthanthecombinednumberofimages tallytocreatethetestmatrixX
µ
inthegalleryandthetestsets. Duetothislimitation,these
methodsonlyworkforsmalltestsets.Hayatetal.[10],[11] Xµ =[x1µx2µx3µ...xMµ ]∈RT×M, (2)
proposedadeeplearningbasedapproachcalledtheAdap-
whereµistheunknownclass.IfX belongstothecthclass
tiveDeepNetworkTemplate(ADNT).Intheirtechnique,a µ
thenitshouldbepossibletorepresenttheimagevectorsof
deepautoencoderisusedtodefineclass-specificmodelsfor
X asalinearcombinationofthegalleryimagesfromthe
trainingsets. Theweightsofanautoencoderareinitialized µ
sameclassi.e.,
with a Gaussian Restricted Boltzmann Machine (GRBM).
Forclassification,eachimageofthetestsetisreconstructed
xm =Q γm, m=1,2,...,M, c=1,2,...,C (3)
usingalearntclass-specificmodelandreconstructionerror µ c c
is used as a measure to identify the test image set. ADNT
whereγm ∈RN×1isavectorofparameters.Fortheunique
hasbeendemonstratedtoachievebetterperformancecom- c
solutionofEquation(3)toexist,theconditionT ≥N must
paredtowell-knownmethods,butitreliesonhandcrafted
hold. Giventhattheconditionholds, γm canbeestimated
LBP features and requires fine tuning of several parame- c
fortestimagevectorxmandregressorQ byusingtheleast
tersforgoodperformance. Moreover, trainingtheADNT µ c
squaresmethod[7],[23],[24]:
requires a large number of images and is computationally
expensive. Our technique reconstructs images in the test γm =(Q(cid:48)Q )−1Q(cid:48)xm, m=1,2,...,M c=1,2,...,C
c c c c µ
image set using LRC from the gallery image matrix and
(4)
is much faster than ADNT both at training and test times. whereQ(cid:48) isthetransposeofQ . Theimagevectorxm can
c c µ
The proposed technique does not have any constraints on
bereconstructedfortheclasscusingtheparametersvector
the number of images in the test set. Moreover, our tech- γmandtheregressorQ :
c c
nique can produce state of the art results using lower res-
olutionimagesandmuchfewertrainingdata, comparedto xm =Q γm, m=1,2,...,M c=1,2,...,C (5)
(cid:98)c c c
othertechniques.
xm =Q (Q(cid:48)Q )−1Q(cid:48)xm (6)
(cid:98)c c c c c µ
3.ProposedTechnique
wherexmisthereconstructedimagevectorforxmfromthe
(cid:98)c µ
regressor Q . xm can also be interpreted as the projection
Let N be the number of gallery images in each unique c (cid:98)c
ofxmonthecthsubspace.
class C of the gallery set Kc. Each image is converted to µ
InsteadofsolvingEquation(6)individuallyforeachim-
grayscale and downsampled to a resolution of a×b to be
representedaskn ∈Ra×b,wherec=1,2,3,...,Candn= age vector xmµ, it can be formulated in the matrix form to
c
efficientlyutilizethecomputationalpowerofmoderncom-
1,2,3,...,N. Each gallery image is transformed through
columnconcatenationtoavectorsuchthatkn ∈ Ra×b → puters:
c
qn ∈ RT×1, where T = ab. Based on the concept that a Xµ =QcΓc, c=1,2,...,C (7)
c
linear subspace is formed by patterns from the same class where Γ ∈ RN×M is a matrix of parameters. Γ can be
c c
[2],aclassspecificmodelQcisconstructedforeachclassc calculatedbyusingtheleastsquareestimation.
byhorizontallyconcatenatingtheimagevectorsofclassc.
Γ =(Q(cid:48)Q )−1Q(cid:48)X , c=1,2,...,C (8)
c c c c µ
Q =[q1q2q3...qN]∈RT×N, c=1,2,3,...,C (1)
c c c c c
X(cid:98)c =QcΓc, c=1,2,...,C (9)
Inthisway, eachclasscisrepresentedbyavectorsub-
space Q called the regressor for class c. Each vector X(cid:98)c =Qc(Q(cid:48)cQc)−1Q(cid:48)cXµ (10)
c
qcn, n=1,2,3,...,N, oftheregressorQcspansasubspace where X(cid:98)c ∈ RT×M is the matrix of reconstructed image
ofRT×1. vectors for X from the regressor Q . The reconstruction
µ c
Let the problem be to classify the unknown class µ of errorbetweeneachtestimagexmandthereconstructedim-
µ
a test image set Yµ with M number of images in one of agex(cid:98)mc iscalculatedusingtheEuclideandistance:
theclassesc = 1,2,3,...,C. Similartothegalleryimages,
each image of the test image set is converted to grayscale dmc =(cid:13)(cid:13)xmµ −x(cid:98)mc (cid:13)(cid:13)2, c=1,2,...,C, m=1,2,...,M
and downsampled to the resolution of a × b to be repre- (11)
sented as ym ∈ Ra×b where µ is the unknown class and We experimented with different voting strategies. Major-
µ
m = 1,2,3,...,M. Each downsampled image is trans- ity voting produced good results on ETH-80 dataset and
formedthroughcolumnconcatenationtoavectorsuchthat UCSD/Honda dataset. Nearest Neighbour Classification
ym ∈ Ra×b → xm ∈ RT×1, where T = ab. The image including 1-NN, 5-NN and 11-NN produced good results
µ µ
on YouTube Celebrity Dataset, UCSD/Honda dataset and Algorithm 1: The Proposed Image Set Classification
MoBo Dataset. However, we required a voting strategy Technique
which performs consistently across all datasets. We also
Input :GalleryimagesetsK ,where
c
experimentedwithdifferentversionsofweightedvoting. In
c=1,2,3,...C. TestimagesetY .
µ
weightedvotingeachimagemofthetestimagesetcastsa
Output:ClassµoftestimagesetY .
voteθmforeachclassctodeterminetheclassofthetestim- µ
c GalleryFormation:
agesetX . WeexperimentedusingtheEuclideandistance,
µ forcin1toC do
the inverse of the Euclidean distance and the square of in-
fornin1toN do
verseEuclideandistanceasweights. However,thebestper- qn ∈RT×1 =downsampleimagestoa×b
formance was achieved when using the exponential of the c
andvectorize,whereT =ab
Euclidean distance in weighted voting. Hence, the weight
end
of vote θcm of each image m is defined by the following Q ∈RT×N =[q1q2q3...qN]
equation: c c c c c
end
θcm =e−αdmc , c=1,2,...,C, m=1,2,...,M (12) Testing:
formin1toM do
where α is a constant. The accumulated weight for each xm ∈RT×1 =downsampleimagestoa×b
µ
classcfromeachimageoftestsetisgivenby: andvectorize,whereT =ab
end
M
Θc = (cid:88) θcm, c=1,2,...,C (13) Xµ ∈RT×M =[x1µx2µx3µ...xMµ ]
forcin1toC do
m=1
formin1toM do
The class c which gets the maximum accumulated weight
γm =(Q(cid:48)Q )−1Q(cid:48)xm
fromalltheimagesxm ofthetestimagesetX isdecided c c c c µ
µ µ xm =Q γm
astheclassofthetestimageset: (cid:98)c c c
(cid:113)
dm = (cid:80)T((xm−xm)2)
µ=argmax(Θ )c=1,2,...,C (14) c µ (cid:98)c
c c θcm =e−αdmc
Algorithm1providestheproposedimagesetclassification end
technique. Θ =(cid:80)M θm
c m=1 c
end
3.1.TheProblemofSingularity
µ=argmax(Θ )
c
c
Asmentionedbefore,forEquation(3)andEquation(7)
to be well conditioned, the total number of pixels T = ab
in downsampled gallery image vectors qn must be greater
c
than or equal to the number of gallery images N in each NotethatEquation(15)isimplementedbeforeanyprepro-
regressorQc i.e., T ≥ N. However, evenifthiscondition cessingandthevaluesinmatrixQc areintherangeof0to
holds, it is possible for regressor Qc to be singular as one 255.Inthisway,themaximumpossiblechangeinthevalue
ormoreoftherowsofQc maycomeouttobelinearlyde- ofanypixelis0.5.Weobservedthattherewasnodeteriora-
pendentonotherrows. Inthiscase, regressorQc iscalled tionintheclassificationaccuracywhenusingthismethod.
rankdeficientduetothefactthatr <T,whereristherank
ofQ . Therefore,itisnotpossibletouseEquation(4)and
c
Equation(8)tocalculatetheparametersvectorγ orparam-
c
etersmatrixΓ . Inthispaperwepresenttwosolutionsfor
c
thisproblem: 3.1.2 BasicSolutionusingQRdecomposition
3.1.1 Perturbation In our second solution, we overcome the problem of sin-
gularity by computing a basic solution for Equation (3) or
ThesingularityoftheregressorQ canbeovercomebyreg-
c
Equation (7) using QR decomposition [21], [25] of the re-
ularizing the regressor Q by adding a small perturbation
c
gressorQ withtheconditionthatthenumberofnon-zero
term[28]. Weempiricallyfoundthatbyaddingamatrixε c
components in the solution vector γ ≤ r, where r is the
withuniformrandomvaluesintherange−0.5≤(cid:15)≤+0.5 c
rank of the regressor Q . This method does not remove
removesthesingularityoftheregressorQ i.e., c
c
thesingularityoftheregressorQ ,however,theresultsob-
c
Q∗ =Q +ε, ε∈RT×Nand∀(cid:15)∈ε, −0.5≤(cid:15)≤+0.5 tainedwiththismethodareaccurateforthepurposeofour
c c
(15) imagereconstructiontechnique.
3.2.FastLinearImageReconstruction
Algorithm2:AlgorithmforFastimagereconstruction
andClassification
A substantial decrease in the processing time can be
Input :GalleryimagesetsK ,where
achievedwhenusingEquations(7), (8), (9)and(10)com- c
c=1,2,3,...C. TestimagesetY .
paredtotheuseofEquations(3),(4),(5)and(6). Thepro- µ
Output:ClassµoftestimagesetY .
cessing time can further be reduced by calculating the in- µ
GalleryFormation:
verse matrix of the regressor Q using the Moore-Penrose
c
forcin1toC do
pseudoinverse[20],[25]atthetimeofgalleryformation. In
fornin1toN do
thisway, thecalculationsattesttimereducetotwomatrix
qn ∈RT×1 =downsampleimagestoa×b
operations (Algorithm 2). Let Q(cid:101)c be the pseudoinverse of c
andvectorize,whereT =ab
theregressorQ calculatedatthetimeofgalleryformation,
c
end
thenEquation(7)canbesolvedattesttimeas:
Q ∈RT×N =[q1q2q3...qN]
c c c c c
Γc =Q(cid:101)cXµ (16) Q(cid:101)c =pseudoinverse(Qc)
end
X(cid:98)c =Qc(Q(cid:101)cXµ) (17) Testing:
formin1toM do
Innumericalanalysistheory, theleastsquaressolutionus- xm ∈RT×1 =downsampleimagestoa×b
µ
ingpseudoinverseisnumericallylessprecisethanusingQR
andvectorize,whereT =ab
decomposition. However,wedidnotobserveanydegrada- end
tionintheaccuracywhenusingthepseudoinverse. Nearly X ∈RT×M =[x1x2x3...xM]
µ µ µ µ µ
twotimesgainincomputationalefficiencywasachievedby
forcin1toC do
thefastlinearimagereconstructionforETH-80dataset(re-
fer to Section 5). The gain in computational efficiency is Γc =Q(cid:101)cXµ
moresubstantialforlargerdatasets. X(cid:98)c =QcΓc
(cid:113)
Dc = (cid:80)T((Xµ−X(cid:98)c)2)
4.ExperimentsandAnalysis Θc =(cid:80)Mm=1e−αDc
end
Extensive experiments were carried out to demonstrate
µ=argmax(Θ )
the performance of our technique. We evaluated our tech- c
c
nique on three video databases, namely CMU Motion of
BodyDataset(CMUMoBo)[8],YoutubeCelebrityDataset
(YTC)[13]andHonda/UCSDDataset[15]forfacerecog-
referenced the recognition results of [4], [9], [11], [12],
nition.ETH-80[16]wasusedforthetaskofobjectrecogni-
[14], [19], [27], [28], [29], [30], [31] and [32] reported in
tion.Weselectedthesedatasetsbecausetheyarecommonly
[11]. DLRC[5]andPLRC[6]followslightlydifferentpro-
usedtoevaluateimagesetclassificationtechniques.
tocolforexperiments. Forthesakeofcompletionwehave
Wecomparedourtechniquewithseveralprominentim-
reported the average results of these techniques, wherever
age set classification methods. These techniques include
available,asreportedintherespectivepapers.
Face Recognition using Temporal Image Sequence (TIS)
[30], Discriminant Canonical Correlation Analysis (DCC)
4.1.CMUMoBoDataset
[14], Manifold-Manifold Distance (MMD) [29], Manifold
Discriminant Analysis (MDA) [27], the Linear version of TheCMUMotionofBodyDatabase(CMUMoBo)[8]
theAffineHullbasedImageSetDistance(AHISD)[4],the contains videos of 25 individuals walking on a treadmill,
ConvexHullbasedImageSetDistance(CHISD)[4],Graph captured from six different viewpoints. Only the videos
EmbeddingDiscriminantAnalysis(GEDA)[9],SparseAp- from the front camera are used for the purpose of image
proximated Nearest Points (SANP) [12], Covariance Dis- set classification. All the subjects except the last one has
criminantLearning(CDL)[28],RegularizedNearestPoints four different videos following different walking patterns.
(RNP)[31],MeanSequenceSparseRepresentationClassi- Thefourwalkingpatternsareslowwalk,fastwalk,inclined
fication (MSSRC) [19], Set to Set Distance Metric Learn- walk and holding a ball while walking. The original pur-
ing (SSDML) [32] and Adaptive Deep Network Template poseofthisdatabasewastoadvancebiometricresearchon
(ADNT)[11]. WealsocomparedourresultswiththeDual human gait analysis [8]. We used the video sequences of
Linear Regression based Classifier (DLRC) [5] and Pair- thefirst24individuals,astheycontainallfourwalkingpat-
wise Linear Regression Model (PLRC) [6]. We followed terns, whichissimilartothepreviousworks[5], [6], [11].
the experimental protocols of Hayat et al. [11] since they Theframesofeachvideowereconsideredasanimageset.
had the best classification accuracy. For comparison, we Similar to [4], [11], [12] and [29], we randomly selected
Methods↓\Datasets→ MoBo YTC Honda ETH-80
TIS[30] 96.81±1.97 50.21±3.59 88.21±3.86 75.50±4.83
DCC[14] 88.89±2.45 51.42±4.95 92.56±2.25 91.75±3.74
MMD[29] 92.50±2.87 54.04±3.69 92.05±2.25 77.50±5.00
MDA[27] 80.97±12.28 55.11±4.55 94.36±3.38 77.25±5.46
AHISD[4] 92.92±2.12 61.49±5.63 91.28±1.79 78.75±5.30
CHISD[4] 96.52±1.18 60.42±5.95 93.62±1.63 79.53±5.32
GEDA[9] 84.86±3.24 52.48±4.45 91.28±5.82 79.50±5.24
SANP[12] 97.64±0.94 65.60±5.57 95.13±3.07 77.75±7.31
CDL[28] 90.00±4.38 56.38±5.31 98.97±1.32 77.75±4.16
RNP[31] 96.11±1.43 65.82±5.39 95.90±2.16 81.00±3.16
MSSRC[19] 97.50±0.88 59.36±5.70 97.95±2.65 90.50±3.07
SSDML[32] 95.14±2.20 66.24±5.21 86.41±3.64 81.00±6.58
DLRC*[5] 91.60±2.78 65.55±5.16 92.31 NA
PLRC*[6] 93.74±4.3 NA 89.74 NA
ADNT[11] 97.92±0.73 71.35±4.83 100.00±0.0 98.12±1.69
Ours 98.33±1.27 66.45±5.07 100.00±0.0 94.75±4.32
Table1.AverageclassificationaccuraciesandstandarddeviationsonCMUMoBo(MoBo)[8],YouTubeCelebrity(YTC)[13],
UCSD/Honda(Honda)[15]andETH-80[16]datasets.Bothalgorithmsfortheproposedtechniquehavethesameaccuracy.
*Indicatesuseofdifferentexperimentalprotocolthantheoneusedfortheothermethods.Averageaccuraciesareshownforthesakeof
completion.NAindicatestheresultsoftherespectivemethodsarenotavailable.
the video of one walking pattern of each individual as the Viola and Jones algorithm [26] failed to detect faces for a
galleryimagesetandtheotherthreewalkingpatternswere largenumberofframes. Therefore, similarto[11], theIn-
considered as the test set. As mentioned in Section 3, the crementalLearningTracker[22]wasusedtotrackthefaces
numberofimagesshouldbelessthanorequaltothenum- invideoclips.Toinitializethetrackerweusedtheinitializa-
ber of pixels in the downsampled images. In practice, the tionparametersprovidedbytheauthorsof[13]1. Although
number of images should be considerably lower than the thecroppedfaceregionwasnotuniformacrossframes,we
number of pixels. We randomly selected a small number decidedtousetheautomaticallytrackedfaceswithoutany
of frames i.e., 50 from each gallery video. The face from refinement. Asproposedin[11],[12],[27],[28],[29],five
eachframewasautomaticallydetectedusingtheViolaand foldcrossvalidationwasusedforexperiments. Thedatset
Jonesfacedetectionalgorithm[26]. Similarto[11],theim- wasdividedintofivefoldswhileminimizingtheoverlapbe-
ageswereresampledtotheresolutionof40×40andcon- tweenthevariousfolds. Eachfoldcontains423videoclips
vertedtograyscale. Histogramequalizationwasappliedto with9videoclipsperindividual.Outof9videoclipsperin-
increase the contrast of images. Different to [5], [6], [11], dividual,threevideoswererandomlyselectedasthegallery
we did not use any LBP features, and performed experi- setwhiletheremainingsixwereusedassixseparatetestim-
mentsonrawimages. Weusedα = 0.2inEquation(12). agesets. Allthetrackedfaceimageswereresampledtothe
The experiments were repeated for 10 times with different resolutionof30×30andconvertedtograyscale,following
randomselectionsforthegalleryandthetestsets. Wealso the protocol of [11]. Histogram equalization was applied
used different random selections of the gallery images in to enhance the contrast of images. For the gallery image
eachroundtomakeourtestingenvironmentmorechalleng- setwerandomlyselected20imagesfromeachofthethree
ing. WeachievedthebestclassificationaccuracyonMoBo galleryvideosperindividualperfold. Ifanygalleryvideo
dataset among all compared techniques. Table 1 provides clip had less than 20 frames, all the images of that video
theaverageaccuracyofourtechniquealongwithacompar- were used for gallery formation. In this way each gallery
isonwithothermethods. set had a maximum of 60 images. We used α = 10.5 in
Equation (12). Our technique achieved the highest accu-
4.2.YouTubeCelebrityDataset racyamongalltheparametricandnon-parametricmethods.
DeepLearningbasedADNT[11]hasabetterclassification
TheYoutubeCelebrity(YTC)Dataset[13]contains1910 accuracy,however,itshouldbenotedthatourmethoduses
video clips of 47 celebrities and politicians. This is the significantlylesstrainingdatacomparedto[11]andismuch
largestdatasetusedforimagesetclassification.Thesenoisy
real world videos, downloaded from YouTube, have low 1http://seqam.rutgers.edu/site/index.php?
resolutionandarerecordedathighcompressionrates. The option=com_content&view=article&id=64&Itemid=80
faster than [11] (refer to Section 5). Moreover our tech- α = 0.2inEquation(12). Werepeatedtheexperiments10
nique does not require any parameter fine tuning or train- timesfordifferentrandomselectionsofgalleryandtestsets.
ingwhichmakesitmoresuitableforreallifeapplications. Theperformanceofourtechniqueiscomparabletothestate
Table1summarizestheaverageaccuraciesofthedifferent oftheartdeeplearningtechnique[11]. Table1summarizes
techniquesonYouTubeCelebritydataset. theresultsofourtechniquecomparedtoothermethods.
4.3.UCSD/HondaDataset 4.5.Experimentsatlowresolution
TheUCSD/HondaDataset[15]consistsof59videosof We carried out further experiments at lower resolutions
20 individuals. The number of videos for each individ- todemonstratetheefficacyofourtechnique. Wealsocom-
ual varies from one to five. The database was originally pared the performance of our technique with ADNT [11]
developed to provide a standard video database to evalu- using low resolution data. We kept all other experimental
ate face tracking and recognition algorithms [15]. All the settings the same as in the previous sections. For [11], we
videos contain significant head rotations and pose varia- used the implementation provided by the authors and kept
tions. Moreover some of the video sequences also contain allprameterssettingsthesameasrecommendedintheirpa-
partial occlusions in some frames. We followed the same per. Theexperimentswererepeated10timeswithrandom
experimentalprotocolas[11],[12],[15],[27]and[29].The selections of gallery and test sets. Table 2 shows the av-
facefromeachframeofvideoswasautomaticallydetected erage classification accuracies which demonstrates the su-
usingViolaandJonesfacedetectionalgorithm[26]. Simi- periorperformanceofourtechniqueatlowresolution. On
larto[11],thedetectedfacesweredownsampledtotheres- CMUMoBodataset[8],theperformanceimprovesatlower
olution of 20×20 and converted to grayscale. Histogram resolution.ForUCSD/Hondadataset[15],theclassification
equalizationwasappliedtoincreasethecontrastofimages. accuracyremainsaperfectscorewiththechangeinresolu-
Theimageswerestandardizedbysubtractingthemeanand tion. OnETH-80Dataset[16],weachievedthebestperfor-
dividing by the standard deviation. We randomly selected manceat20×20resolution. Thisisduetothefactthatat
onevideofromeachofthe20individualsasthegalleryim- 15×15resolution,thenumberofgalleryimagesisnearly
agesetwhiletheremaining39videoswereusedasthetest equal to number of pixels (refer to Section 3 for details).
image sets. In order to keep the number of gallery images At 15×15 resolution, we are able to achieve an average
considerablylessthanthenumberofpixels(refertosection classification accuracy of 95.25% by reducing the number
3),werandomlyselectedasmallnumberofframesi.e.,50 ofgalleryimages. Overall,thereisnosignificantchangein
from each gallery video. We used α = 0.2 in Equation theclassificationaccuracyofourtechniquewiththechange
(12). Toimprovetheconsistencyinscoreswerepeatedthe inresolution. ComparedtoADNT[11],ourtechniquecon-
experiment 10 times with a different random selection of sistently achieved better performance using low resolution
galleryimages,galleryimagesetsandtestimagesets. Our data. This shows that our technique is more suitable to
technique achieved a perfect classification accuracy while applications where the data is of very low resolution e.g.,
using a significantly less number of gallery images. Table CCTVsurveillance.
1 summarized the average identification rates of our tech-
niquecomparedtootherimagesetclassificationtechniques. 5.ComputationalTimeAnalysis
4.4.ETH-80Dataset
Theproposedtechniqueachievesthefastesttimingper-
The ETH-80 dataset [16]2 consists of eight object cat- formance as compared to other techniques. Table 3 shows
egories including fruits, animals and vehicles. The object the training time for various methods and the test time re-
categories are apples, pears, tomatoes, cows, dogs, horses, quiredtoclassifyanimagesetontheETH-80datasetusing
carsandcups. Eachobjectcategoryhaseightdifferentim- a modern CPU with 8 GB RAM. The proposed technique
agesets. Eachimagesetconsistsof41imagesoftheobject requires no training. Although the proposed technique re-
takenfromdifferentviewangles. Thecroppedimagescon- constructs each image of the test image set from all the
tainingonlytheobjectwithoutanyborderareawereused. galleryimagesets,butduetotheefficientmatrixrepresenta-
The images were resized to the resolution of 32 × 32 to tion(refertosection3andsection3.2),theachievedtiming
followtheprotocolof[11]. Theimageswereconvertedto efficiencyissuperiortotheothermethods.
grayscale and were standardized by subtracting the mean
6.Discussion
and dividing by the standard deviation. Similar to [11],
[14],[27]and[28],fiveimagesetsofeachobjectcategory
OurproposedtechniqueaswellasDLRC[5]andPLRC
arerandomlyselectedasthegallerysetwhiletheotherfive
[6] use linear regression for classification. However, our
areconsideredtobeindependenttestimagesets. Weused
technique is remarkably different from DLRC and PLRC.
2http://www.d2.mpi-inf.mpg.de/Datasets/ETH80 DLRCconsidersbothtrainingandtestimagessetsassub-
Dataset↓ Method↓ Resolutionusedby[11] 20×20Resolution 15×15Resolution
ADNT[11] 97.92±0.73 91.81±2.40 90.56±3.13
MoBo
Ours 98.33±1.27 98.75±1.38 99.31±1.18
ADNT[11] 100.00±0.00 100.00±0.00 99.74±0.81
Honda
Ours 100.00±0.00 100.00±0.00 100.00±0.00
ADNT[11] 98.12±1.69 88.75±6.26 90.25±4.63
ETH-80
Ours 94.75±4.32 95.50±4.04 92.75±6.39*
Table2.AverageclassificationaccuraciesandstandarddeviationsonthelowresolutionsofourtechniquecomparedwithADNT[11].
*Theslightdecreaseinperformanceisduetothenearlyequalnumberofgalleryimagesandthenumberofpixels(refertoSection3fordetails).
spaces of a high dimensional space and uses the distance TotalTraining
TestTimeper
between the test and training image sets to determine the Methods↓ Time
imageset(seconds)
class of the test image set. To determine the distance be- (seconds)
tweensubspaces,DLRCusesthelastimageofeachimage TIS[30] NR 0.045
set along with the variations between the training and test DCC[14] 13.36 0.311
imagesets[6]tosolvealinearregressionproblem. PLRC MMD[29] NR 8.43
isanextensionofDLRC.InPLRCinsteadofthelastimage MDA[27] 1.22 0.005
of each image set, the mean image is used along with the AHISD[4] NR 0.095
conceptofrelatedandunrelatedsubspaces[6].BothDLRC CHISD[4] NR 0.213
and PLRC have the limitation that the combined number GEDA[9] 2.7 0.068
of images in the test and the training image sets should be SANP[12] NR 105.7
much less than the number of features in the feature vec-
CDL[28] 76.21 1.40
tors. In addition to the gallery image sets, they also use
RNP[31] NR 0.027
test image sets as regressors, which render them prone to
MSSRC[19] NR 4.78
theproblemofrankdeficientmatrixattesttime. Moreover
SSDML[32] 21.92 0.577
theyuseLBPfeatures[18]forcertaindatasets. Therefore,
ADNT[11] 278.8 0.026
theperformanceofthesemethodsusingrawimagescannot
Ours NR 0.0046
begeneralized.
Ours(Fast) NR 0.0028
Our technique is quite different from DLRC and PLRC Table 3. Computational Time Analysis on ETH-80 dataset. NR
as we treat each image in the test image set independently showsthatthemethoddoesnotrequiretraining.
and consider them as points in a high dimensional space.
Wereconstructeachtestimagefromthegallerysubspaces
and use weighted voting with the Euclidean distances be- CelebritydatasetandtheETH-80dataset. However,ADNT
tween the original and reconstructed test images. The use needs a lot of training data and relies on handcrafted LBP
ofweightedvotingincreasestherobustnessofoursystemto features. ADNTusesRestrictedBoltzmanMachineforpa-
anynoiseandoutliersinthetestimageset. Ourtechnique rameter initialization and requires a lot of fine tuning. On
does not impose any constraints on the number of images the other hand, our technique uses only a fraction of the
in the test set and can work with any number of test im- training data and achieves comparable results, using only
ages. Moreover, we performed all of our experiments on therawimages. Moreover,ourtechniquehasproducedsu-
rawimagestodemonstratethegeneralizabilityofourtech- periorresultsatlowerresolutions,comparedtoADNT,and
nique.IncontrasttoDLRCandPLRC,onceanysingularity is ten times faster than ADNT at test time. Our technique
isremovedintheregressorQ atthetimeofthegalleryfor- canalsobeeasilygeneralizedtonewdata.Thecapabilityto
c
mation (refer to section 3.1), our technique is immune to workwithlesstrainingdataandatlowresolutiondeemsour
the problem of rank deficient matrix at test time. This is techniquesuitableforscenarioswhereonlyscarcetraining
due to the fact that the test image set is not used as a re- dataisavailableandwherefastdecisionsarerequired.
gressor.Ourtechniquecanprocesswholeimagesetssimul-
taneouslyandalsohasthecapabilitytoprocessoneimage 7.Conclusion
at a time and update the class decision in real time which
makes it suitable for live video surveillance (e.g., CCTV). In this work, a novel image set classification technique
The accuracy of the proposed technique is superior to all isproposed. Theproposedtechniqueuseslinearregression
parametricandnon-parametricmethods. Thedeeplearning toreconstructimagesofthetestimagesetfromgalleryim-
techniqueADNT[11]hasabetteraccuracyontheYoutube agesetsandusestheaccumulativeweightedreconstruction
errortodecidefortheclassofthetestimageset. Thetech- [12] Y. Hu, A. S. Mian, and R. Owens. Face recognition us-
nique requires less training data compared to other image ingsparseapproximatednearestpointsbetweenimagesets.
set classification methods and can work effectively at very IEEEtransactionsonPatternAnalysisandMachineIntelli-
low resolution. Extensive experimental analysis has been gence,34(10):1992–2004,2012.
presentedonanumberofpopularandchallengingdatasets [13] M. Kim, S. Kumar, V. Pavlovic, and H. Rowley. Face
trackingandrecognitionwithvisualconstraintsinreal-world
to demonstrate the superior performance of our technique.
videos. In2008IEEEConferenceonComputerVisionand
Throughtheefficientmatriximplementation, theproposed
PatternRecognition(CVPR),pages1–8.IEEE,2008.
techniqueachievesthefastestperformancetime. Thetech-
[14] T.-K.Kim,J.Kittler,andR.Cipolla.Discriminativelearning
nique can easily be scaled from processing one frame at a
andrecognitionofimagesetclassesusingcanonicalcorrela-
time(forlivevideoacquisition)toprocessingallofthetest
tions. IEEETransactionsonPatternAnalysisandMachine
dataatonce(forfasterperformance).Allthesefactorsmake Intelligence,29(6):1005–1018,2007.
ourtechniqueidealforimagesetclassificationapplications. [15] K.-C. Lee, J. Ho, M.-H. Yang, and D. Kriegman. Video-
basedfacerecognitionusingprobabilisticappearanceman-
References ifolds. In 2003 IEEE Conference on Computer Vision and
PatternRecognition(CVPR),volume1,pagesI–313.IEEE,
[1] O.Arandjelovic,G.Shakhnarovich,J.Fisher,R.Cipolla,and
2003.
T.Darrell. Facerecognitionwithimagesetsusingmanifold
[16] B. Leibe and B. Schiele. Analyzing appearance and con-
densitydivergence. In2005IEEEConferenceonComputer
tourbasedmethodsforobjectcategorization. In2003IEEE
Vision and Pattern Recognition (CVPR), volume 1, pages
Conference on Computer Vision and Pattern Recognition
581–588.IEEE,2005.
(CVPR),volume2,pagesII–409.IEEE,2003.
[2] R.BasriandD.W.Jacobs. Lambertianreflectanceandlin-
[17] I. Naseem, R. Togneri, and M. Bennamoun. Linear re-
ear subspaces. IEEE transactions on Pattern Analysis and
gression for face recognition. IEEE Transactions on Pat-
MachineIntelligence,25(2):218–233,2003.
ternAnalysisandMachineIntelligence,32(11):2106–2112,
[3] P.N.Belhumeur,J.P.Hespanha,andD.J.Kriegman.Eigen- 2010.
facesvs.fisherfaces: Recognitionusingclassspecificlinear [18] T.Ojala, M.Pietikainen, andT.Maenpaa. Multiresolution
projection. IEEETransactionsonPatternAnalysisandMa- gray-scale and rotation invariant texture classification with
chineIntelligence,19(7):711–720,1997. localbinarypatterns.IEEETransactionsonPatternAnalysis
[4] H.CevikalpandB.Triggs. Facerecognitionbasedonim- andMachineIntelligence,24(7):971–987,2002.
agesets. In2010IEEEConferenceonComputerVisionand [19] E. G. Ortiz, A. Wright, and M. Shah. Face recognition
PatternRecognition(CVPR),pages2567–2573.IEEE,2010. in movie trailers via mean sequence sparse representation-
basedclassification. In2013IEEEConferenceonComputer
[5] L.Chen. Duallinearregressionbasedclassificationforface VisionandPatternRecognition(CVPR),pages3531–3538,
clusterrecognition. In2014IEEEConferenceonComputer 2013.
VisionandPatternRecognition(CVPR),pages2673–2680. [20] R.Penrose. Onbestapproximatesolutionsoflinearmatrix
IEEE,2014. equations. InMathematicalProceedingsoftheCambridge
[6] Q. Feng, Y. Zhou, and R. Lan. Pairwise linear regression PhilosophicalSociety,volume52,pages17–19.Cambridge
classificationforimagesetretrieval. In2016IEEEConfer- UnivPress,1956.
enceonComputerVisionandPatternRecognition(CVPR), [21] W.H.Press. Numericalrecipes3rdedition: Theartofsci-
pages4865–4872,2016. entificcomputing. Cambridgeuniversitypress,2007.
[7] J.Friedman, T.Hastie, andR. Tibshirani. The elementsof [22] D.A.Ross,J.Lim,R.-S.Lin,andM.-H.Yang. Incremental
statistical learning, volume 1. Springer series in statistics learningforrobustvisualtracking. InternationalJournalof
Springer,Berlin,2001. ComputerVision,77(1-3):125–141,2008.
[8] R. Gross and J. Shi. The cmu motion of body (mobo) [23] T.P.Ryan. Modernregressionmethods,volume655. John
database. 2001. Wiley&Sons,2008.
[9] M.T.Harandi, C.Sanderson, S.Shirazi, andB.C.Lovell. [24] G.A.SeberandA.J.Lee. Linearregressionanalysis,vol-
Graph embedding discriminant analysis on grassmannian ume936. JohnWiley&Sons,2012.
manifoldsforimprovedimagesetmatching. In2011IEEE [25] J.StoerandR.Bulirsch. Introductiontonumericalanalysis,
Conference on Computer Vision and Pattern Recognition volume12. SpringerScience&BusinessMedia,2013.
(CVPR),pages2705–2712.IEEE,2011. [26] P. Viola and M. J. Jones. Robust real-time face detection.
[10] M.Hayat,M.Bennamoun,andS.An. Learningnon-linear International Journal of Computer Vision, 57(2):137–154,
reconstructionmodelsforimagesetclassification. In2014 2004.
IEEEConferenceonComputerVisionandPatternRecogni- [27] R. Wang and X. Chen. Manifold discriminant analysis.
tion(CVPR),pages1907–1914,2014. In2009IEEEConferenceonComputerVisionandPattern
[11] M.Hayat,M.Bennamoun,andS.An. Deepreconstruction Recognition(CVPR),pages429–436.IEEE,2009.
models for image set classification. IEEE transactions on [28] R.Wang,H.Guo,L.S.Davis,andQ.Dai. Covariancedis-
PatternAnalysisandMachineIntelligence, 37(4):713–727, criminativelearning:Anaturalandefficientapproachtoim-
2015. agesetclassification.In2012IEEEConferenceonComputer
VisionandPatternRecognition(CVPR),pages2496–2503.
IEEE,2012.
[29] R.Wang,S.Shan,X.Chen,andW.Gao.Manifold-manifold
distance with application to face recognition based on im-
ageset. In2008IEEEConferenceonComputerVisionand
PatternRecognition(CVPR),pages1–8.IEEE,2008.
[30] O. Yamaguchi, K. Fukui, and K.-i. Maeda. Face recogni-
tionusingtemporalimagesequence. InAutomaticFaceand
GestureRecognition,1998.Proceedings.ThirdIEEEInter-
nationalConferenceon,pages318–323.IEEE,1998.
[31] M.Yang,P.Zhu,L.VanGool,andL.Zhang. Facerecogni-
tionbasedonregularizednearestpointsbetweenimagesets.
InAutomaticFaceandGestureRecognition(FG),201310th
IEEEInternationalConferenceandWorkshopson,pages1–
7.IEEE,2013.
[32] P. Zhu, L. Zhang, W. Zuo, and D. Zhang. From point to
set: Extendthelearningofdistancemetrics. In2013IEEE
Conference on Computer Vision and Pattern Recognition
(CVPR),pages2664–2671,2013.