Dimensionality-Dependent Generalization Bounds for $k$-Dimensional Coding Schemes

0.84 MB·

by  Tongliang Liu

1
6
1
0
2
Dimensionality-Dependent Generalization Bounds

r
p
A for k-Dimensional Coding Schemes

3
2

]
L
1 1 2
M Tongliang Liu , Dacheng Tao , Dong Xu
.
t 1
a QCIS,UniversityofTechnologySydney. [email protected];[email protected]
t
s
[ 2
SchoolofComputerEngineering,NanyangTechnologicalUniversity. [email protected]

2
v
8
3
2
0 Keywords: Generalization bound, Bennett type inequality, covering number, k-
0
.
1 dimensional coding schemes, non-negative matrix factorization, dictionary learning,
0
6
1 sparsecoding,k-means clusteringand vectorquantization.
:
v
i
X
r
a
Abstract
Thek-dimensionalcodingschemesrefertoacollectionofmethodsthatattempttorep-
resentdatausingasetofrepresentativek-dimensionalvectors,andincludenon-negative
matrix factorization, dictionary learning, sparse coding, k-means clustering and vector
quantization as special cases. Previous generalization bounds for the reconstruction
error of the k-dimensional coding schemes are mainly dimensionality-independent. A
major advantage of these bounds is that they can be used to analyze the generalization
errorwhendataismappedintoaninfinite-orhigh-dimensionalfeaturespace. However,
many applications use finite-dimensionaldata features. Can we obtain dimensionality-
dependent generalization bounds for k-dimensional coding schemes that are tighter
than dimensionality-independent bounds when data is in a finite-dimensional feature
space? The answer is positive. In this paper, we address this problem and derive a
dimensionality-dependent generalization bound for k-dimensional coding schemes by
bounding the covering number of the loss function class induced by the reconstruction
error. Theboundisoforder (mkln(mkn)/n)λn ,wheremisthedimensionoffea-
O
(cid:16) (cid:17)
tures, k is the number of the columns in the linear implementation of coding schemes,
n is the size of sample, λ > 0.5 when n is finite and λ = 0.5 when n is infinite. We
n n
showthatourboundcanbetighterthanpreviousresults,becauseitavoidsinducingthe
worst-case upper bound on k of the loss function. The proposed generalization bound
isalsoappliedtosomespecificcodingschemestodemonstratethatthedimensionality-
dependent bound is an indispensable complement to the dimensionality-independent
generalizationbounds.
1 Introduction
The k-dimensional coding schemes (Maurer&Pontil, 2010) are abstract and general
descriptions of a collection of methods, all of which encode a data point x as a
∈ H
representative vector y Rk by a linear map T, where denotes the Hilbert space.
∈ H
2
Thesecodingschemescan beformulated asfollows:
yˆ= argmin x Ty 2,
y Y k − k
∈
where Y Rk is called the codebook and the linear map T Rm k is called the
×
⊆ ∈
implementation of the codebook. The implementation projects the codebook back to
thedatasourcespace. Thedimensionofadatapointxcanbeeitherfiniteorinfinite. In
thispaper,weconsiderthedataashavingfinitedimensionsoffeatures,thatis = Rm.
H
Each data point in can be exactly or approximately reconstructed by a code y in
H
thecodebook. Thereconstructionerror ofadatapointxisdefined as
f (x) = min x Ty 2. (1)
T
y Y k − k
∈
Thefunction f (x), whosevariablesare x and T, is also called the lossfunction. Non-
T
negative matrix factorization (NMF) (see, e.g., Leeand Seung, 1999; Fe´votteet al.,
2009), dictionary learning (see, e.g., Chen et al., 1999; Ivana&Pascal, 2011), sparse
coding (see, e.g., Olshausen&Field, 1996; Amiri& Haykin, 2014), k-means cluster-
ing (see, e.g., MacQueen et al., 1967; Anderberg, 1973) and vector quantization (see,
e.g., Gray, 1984; Schneider et al., 2009a) are specific forms of k-dimensional coding
schemes, because they share the same form of the reconstruction error as equation
(1). They have achieved great successes in the fields of pattern recognition and ma-
chinelearningfortheirsuperiorperformancesonabroadspectrumofapplications(see,
e.g., Pehlevanet al., 2015; Mairalet al., 2012; Huntet al., 2012; Wrightet al., 2009;
Schneideret al.,2009b;Dhillonetal.,2007;Quirogaetal.,2004;Kanungoet al.,2002;
Abbott&Dayan, 1999).
Anycoding schemeshouldfind aproperimplementationT. A natural choiceforT
3
istheonethatminimizestheexpected reconstructionerror
R(T) = f (x)dρ(x) = f (x)p(x)dx,
T T
Zx Zx
where ρ(x) is a Borel measure of the data source, and p(x) is the probability den-
sity function. However, in most cases, p(x) is unknown, and R(T) cannot be directly
minimized. An alternative approach is the empirical risk minimization (ERM) method
(Vapnik,2000;Cucker& Smale,2002). Givenafinitenumberofindependentandiden-
tically distributed observations x ,...,x Rm, the empirical reconstruction error
1 n
∈
withrespect toT isdefined as
n
1
R (T) = f (x ).
n T i
n
i=1
X
The ERM method searches for a T that minimizesR (T), and in thehope that R(T )
n n n
has asmalldistancetotheexpected reconstructionerror R(T ), where
∗
T = argminR(T),
∗
T
∈T
and denotes aparticularclass oflinearoperatorsT.
T
A probabilisticboundon thedefect
sup R(T) R (T)
n
| − |
T
∈T
is called the generalization (error) bound. This paper focuses on this error bound in
the framework of k-dimensional coding schemes. Although different restrictions are
imposedon the choices of and Y fordifferent concrete forms of k-dimensionalcod-
T
ing schemes (for example, NMF requires both and Y to be non-negative,and sparse
T
codingrequiressparsityinY),theyarecloselyrelated. Forexample,Dinget al.(2005)
4
showed that NMF with orthogonal (y ,...,y ) is identical to k-means clustering of
1 n ⊤
x ,...,x . Sincethesedifferentformsofk-dimensionalcodingschemesareclosely
1 n
{ }
related, analyzingthegeneralizationboundstogetherinthiscontexthastheadvantages
ofexploitingthecommonpropertiesand mutualcross-fertilization.
1.1 Related work
Maurer& Pontil (2010) and Gribonvalet al. (2015) have performed the only known
theoreticalanalysesonthegeneralizationerrorintheframeworkofk-dimensionalcod-
ing schemes. Other works have concentrated only on specific k-dimensional coding
schemes. Since some previous works have studied consistency performance, which
considersthequantityR(T ) R(T )oftherelatedERM-basedalgorithms,wedemon-
n ∗
−
strate the relationship between the generalization error and consistency performance
here:
R(T ) R(T )
n ∗
−
= R(T ) R (T )+R (T ) R (T )+R (T ) R(T )
n n n n n n ∗ n ∗ ∗
− − −
R(T ) R (T )+R (T ) R(T )
n n n n ∗ ∗
≤ − −
2sup R(T) R (T) .
n
≤ | − |
T
∈T
Thus,analyzingthegeneralizationerrorprovidesanapproachforanalyzingtheconsis-
tency performance, and theconsistencyperformance providesdirections to generaliza-
tion error analysis. We review the generalization error and consistencyperformance of
k-dimensionalcodingschemes together:
Non-negativematrixfactorization(NMF).Theonlyknowngeneralizationbounds
•
5
ofNMFaredevelopedby Maurer& Pontil(2010)and Gribonvalet al. (2015).
Dictionary learning. Maurer& Pontil (2010) have developped dimensionality-
•
independentgeneralizationbounds. Vainsencheret al.(2011)andGribonvalet al.
(2015)havestudiedthedimensionality-dependentgeneralizationbounds.
Sparse coding. A generalization bound for sparse coding was first derived by
•
Maurer&Pontil (2010), and subsequently extended by Xu &Lafferty (2012),
Mehta& Gray(2013),Maurer etal.(2013),andGribonvalet al.(2015). Maureret al.
(2013) derived a faster convergence rate upper bound of the consistency perfor-
manceinatransfer learningsetting.
K-means clustering and vector quantization. Consistency performances of k-
•
means clusteringand vectorquantizationhavemostlybeen studied for = Rm.
H
Asymptoticand non-asymptoticconsistencyperformances havebeen considered
byPollard(1982),Chou(1994),Linderet al.(1994),Bartlett et al.(1998),Linder
(2000), Antoset al. (2005), Antos (2005) and Levrard et al. (2013). Recently,
Biau et al. (2008), Maurer& Pontil (2010) and Levrardet al. (2015) developed
dimensionality-independentgeneralizationboundsfork-means clustering.
Weareawarethatthesespecificformsofk-dimensionalcodingschemeshavemanyap-
plications for finite-dimensional data, and only a few dimensionality-dependent meth-
ods have been developed to analyze the generalization bounds for all these coding
schemes.
In this paper, we develop a dimensionality-dependent method to analyze the gen-
eralization bounds for the framework of k-dimensional coding schemes. Our method
6
is based on Hoeffding’sinequality (Hoeffding, 1963) and the Bennett type inequalities
(Boucheron et al., 2013), and directly bounds the covering number of the loss function
class induced by the reconstruction error, which avoids inducing the worst-case upper
bound on k of the loss function. Our method allows a generalization bound of order
((mkln(mkn)/n)γn), where γ is muchbiggerthan0.5 when nissmall,which del-
n
O
icately describes the non-asymptotic behavior of the learning process. However, when
n goes to infinity, γ approaches to 0.5. The obtained dimensionality-dependent gen-
n
eralization bound can be much tighter than the previous ones when the number k of
columns of the implementation is larger than the dimensionality m, which could often
happen for dictionary learning, sparse coding, k-means clustering and vector quanti-
zation. We therefore obtain state-of-the-art generalization bounds for NMF, dictionary
learning,sparsecoding,k-means clusteringandvectorquantization.
The remainder of the paper is organized as follows. We present our motivation in
Section 2 and main results in Section 3. In Section 4, our results are applied to spe-
cific coding schemes and are empirically compared with state-of-the-art generalization
bounds. Weproveourresultsin Section 5and concludethepaperin Section 6.
2 Motivation
We first introduce the dimensionality-independent generalization bounds and demon-
stratewhyourdimensionality-dependentboundcomplementsthem.
Assume that data points are drawn from a Hilbert space with distributionµ. For
H
any r 0, let (r) denote the set of probability distributions on supported on the
≥ P H
7
closed ball of radius r centered at the origin. In other words, µ (r) means that
∈ P
P x r = 1. Let be bounded in the operator norm, that is for every T , it
{k k ≤ } T ∈ T
holds that Tv c for all v with v 1. Then, we also have that the columns of T
k k ≤ k k ≤
areboundedas Te c,i = 1,...,k,where e 1 i k istheorthonormalbasis
i i
k k ≤ { | ≤ ≤ }
ofRk.
ThefollowingtwotheoremsareequivalenttothemaintheoremsprovedbyMaurer&Pontil
(2010), but are represented in a different way. They are dimensionality-independent
generalization bounds obtained in the frame of the k-dimensional coding schemes.
They exploited the Rademacher complexity technique (Bartlett &Mendelson, 2003)
whichissuitableforderivingdimensionality-independentbounds(seeBiau et al.,2008).
Theorem 1 Assume that µ (r) and Y is a closed subset of the unit ball of Rk,
∈ P
and that there is c 0 such that for all T , Te c,i = 1,...,k. Suppose
i
≥ ∈ T k k ≤
that the reconstruction error functions f for T have a range contained in [0,b].
T
∈ T
For any δ (0,1), with probability at least 1 δ in the independently observed data
∈ −
x ,...,x µ,we have
1 n
∼
π 8ln2/δ
sup R(T) R (T) (4crk+2c2k2) +b .
n
| − | ≤ n n
T r r
∈T
Remark1 Thedimensionality-independentgeneralizationboundinTheorem1isvalu-
ablebecauseit showsa convergencerateof orderO( 1/n).
p
Theorem 2 Assume that µ (r) and = sup sup Ty , and that the
∈ P kTkY T∈T y∈Y k k
reconstruction error functions f for T have a range contained in [0,b]. For
T
∈ T
any δ (0,1), with probability at least 1 δ in the independently observed data
∈ −
8
x ,...,x µ,we have
1 n
∼
sup R(T) R (T)
n
| − |
T
∈T
ln2/δ bk ln(16n 2 ) 4+4 +√8πrk
b + kTkY + kTkY kT kY .
≤ 2n 2 n √n
r r
If isfinitedimensional,theaboveresultwillbeimprovedto
H
sup R(T) R (T)
n
| − |
T
∈T
ln2/δ b mkln(16n 2) 4+4 +√8πrk
b + kTkY + kT kY kTkY .
≤ 2n 2 n √n
r r
Remark2 The condition that Y is a closed subset of the unit ball of Rk can be easily
achieved by controlling the upper bound of columns of T, because there is a trade-off
between theboundsofcolumnsof T and theentriesof y Y.
∈
Remark3 We note that Theorems 1 and 2 are more complicated than the original
results presented in (Maurer &Pontil, 2010). This is because we have removed the re-
strictions that c 1 and 1, which are required to simplify their results, to
Y
≥ kT k ≥
revealtheintrinsicrelationshipsbetweentheorderofk andtheRademachercomplexi-
ties(discussedbelow). TheproofmethodsofTheorems1and2inthispaperareexactly
thesameas thosepresented byMaurer &Pontil(2010).
We notethatify is intheunitballofRk, then
f (x) = min x Ty 2 min x 2 + Ty 2 r2 + min Ty 2
T
y Rk×1k − k ≤ y Rk×1 k k k k ≤ y Rk×1k k
∈ ∈ ∈
k (cid:0) (cid:1) k
= r2 + min y Te ,y Te r2 + min y Te y Te
i i j j i i j j
y Rk×1 h i ≤ y Rk×1 k kk k
∈ i,j ∈ i,j
X X
r2 +c2k2,
≤
9
wherer,theupperboundofthedatapoint,canbereduced bynormalization. However,
k isafixedinteger,whosevalueisusuallylargeinpractice. Thus,c2k2 isthedominant
factorintheupperboundoff . Itisevidentthatf hastheworst-caseupperboundon
T T
k of order (k2), i.e., the dependency w.r.t. k of the upper bound of f has the worst
T
O
case order (k2). However, for some special forms of k-dimensional coding schemes,
O
the upperbound of f has a very small order about k. Taking NMF as an example, the
T
orderaboutk is zero because
f (x) = min x Ty 2 x 2 + T0 2 r2.
T
y Rk k − k ≤ k k k k ≤
∈ +
It is evident that the term 2c2k2 π/n in Theorem 1 has the same order as that of
p
the worst-case upper bound on k of f . It will therefore be loose for some specific
T
k-dimensional coding schemes. Maurer& Pontil (2010) introduced the proof method
of Theorem 2 to overcome this problem; however, the term rk 8π/n implies
Y
kTk
p
that the problem is only partially solved, because rk represents the worst-case upper
bound on k of √f (details can be found in the proof therein). For example, in NMF,
T
the term rk 8π/n is of order ( k3/n) (discussed below in Remark 4). The
Y
kTk O
p p
dimensionality-dependent bound in Theorem 2 faces the same problem because the
proof method computes the Rademacher complexity, corresponding to which part the
obtainedboundisdimensionality-independentandinvolvestheworst-caseupperbound
on k of√f .
T
We try to avoid the aforementioned worst case by employing a covering number
method to measure the complexity of the induced loss function class F = f T
T
T { | ∈
. However,inoursetting,thedimensionalitymofdataspacemustbefinite.
T }
10