Discriminative Training of Deep Fully-connected Continuous CRF with Task-specific Loss

9.5 MB·

by  Fayao Liu

Discriminative Training of Deep Fully-connected Continuous CRFs
with Task-specific Loss
Fayao Liu, Guosheng Lin, Chunhua Shen∗
School of Computer Science, University of Adelaide, Australia
January 29, 2016
6 Abstract otalroleinthecomputervisioncommunity,andarefun-
1
damentaltomanyhigh-leveltasks,likerecognition,scene
0
Recent works on deep conditional random fields (CRFs) understanding, 3D modeling. In the central of many
2
have set new records on many vision tasks involv- pixellabellingtasksstandsthefeatureengineering,which
n ing structured predictions. Here we propose a fully- has a significant impact on the final results. However,
a
J connected deep continuous CRFs model for both dis- designing effective and highly discriminative features re-
crete and continuous labelling problems. We exemplify mainsachallenge. Traditionaleffortshavebeenfocusing
8
2 the usefulness of the proposed model on multi-class se- on designing hand-crafted features, e.g., texton, HOG,
mantic labelling (discrete) and the robust depth estima- SIFT, GIST, etc. Later on, unsupervised feature learn-

] tion(continuous)problems. Inourframework,wemodel ing [1] has been introduced and showed promises. Re-
V
both the unary and the pairwise potential functions as cently,supervisedfeaturelearning,i.e.,deepCNNshave
C deep convolutional neural networks (CNNs), which are achieved tremendous success in various domains includ-
s. jointly learned in an end-to-end fashion. The proposed ing image classification [2], pose estimation [3], seman-
c method possesses the main advantage of continuously- tic labelling [4–6], depth estimation [7,8], etc. Adapting
[
valued CRFs, which is a closed-form solution for the CNNs for pixel labelling tasks have since attracted a lot

1 Maximum a posteriori (MAP) inference. of attention. However, several issues need to be resolved
v Tobetteradapttodifferenttasks, insteadofusingthe inthisprocedure,e.g.,down-sampledcoarsepredictions,
9
commonly employed maximum likelihood CRFs param- non-sharp boundary transitions.
4
eterlearningprotocol,weproposetask-specificlossfunc-
6 Besides the feature representation, another essential
tions for learning the CRFs parameters. It enables di-
7
factortobeconsideredistheappearanceandspatialcon-
0 rect optimization of the quality of the MAP estimates
sistency,whichistypicallycapturedbygraphicalmodels.
. during the course of learning. Specifically, we optimize
1 Probabilistic graphical models, e.g., CRFs [9], have long
0 the multi-class classification loss for the semantic la-
been shown as a fundamental tool in computer vision,
6 belling task and the Turkey’s biweight loss for the ro-
and have been widely applied to structured prediction
1 bust depth estimation problem. Experimental results on
: problems,likeimagedenoising,semanticlabelling,depth
v thesemanticlabellingandrobustdepthestimationtasks
estimation, etc. Recently, continuous CRFs have shown
i demonstrate that the proposed method compare favor-
X benefits of efficient learning and inference, while being
ably against both baseline and state-of-the-art methods.
r as powerful as discrete CRFs models [8,10–13]. In this
a In particular, we show that although the proposed deep
paper, we present a general deep CRFs learning frame-
CRFs model is continuously valued, with the equipment
workforbothdiscreteandcontinuouslabellingproblems,
of task-specific loss, it achieves impressive results even
which combines the benefits of CNNs and continuous
on discrete labelling tasks.
CRFs.
Whilethemaximumlikelihoodlearningisdominantin
1 Introduction
learning CRFs parameters, it is argued being beneficial
to directly take into account the quality of the MAP es-
Recent works on combining conditional random fields
timates during the course of learning. Samuel et al. [11]
(CRFs) and deep convolutional neural networks (CNNs)
pointed out that if one’s goal is to use MAP estimates
have set new records on many vision tasks involving
and evaluate the results using other criteria, then opti-
structured predictions. Pixel-level labelling tasks gen-
mization of the MRFs parameters using a probabilistic
erally refer to assigning a discrete or continuous label to
criterion, like maximum likelihood, may not necessar-
each pixel in an image, with typical examples being se-
ily lead to optimal results. Instead, a better strategy
manticlabelling,anddepthestimation. Theyplayapiv-
is to find the parameters such that the quality of the
∗Correspondingauthor(e-mail: [email protected]). MAP estimates are directly optimized against a task-
specific empirical risk. For example, in image segmen- 1.1 Related work
tation, the most direct criterion of a good segmenta-
Inthissection,wereviewsomemostrecentprogressesin
tion mask is the global accuracy or the Jaccard index
CNNs based methods for semantic labelling and depth
(intersection-over-union score). On the other hand, in
estimation tasks, which are closely related to our work.
many practical applications, it may not be necessary to
model the full conditional distribution of labels, given
CNNs for semantic labelling The most straightfor-
observations. Instead,modelingthemarginalconditional
ward way of applying CNNs for semantic labelling is to
likelihoodismorepreferable. In[14],Domkearguedthat
design fully convolutional networks to directly output
themarginalbasedlossgenerallyyieldsbettermaximum
the prediction map, e.g., [4,6]. These methods do not
posteriormarginal(MPM)inference. Thereforeanalter-
involveoptimizedstructuredloss,whileonlyconsiderin-
nativetothemaximumlikelihoodlearningistominimize
dependent classification. A more appealing direction is
the empirical risk loss. In [15], Kakade et al. propose to
tocombineCNNswithgraphicalmodels,whichexplicitly
maximize the label-wise marginal likelihood rather than
incorporates structured constraints. A simple strategy
thejointlabellikelihoodforsequencelabellingproblems.
to do so is to incorporate consistency constraints into a
Here we analogously propose to maximize the label-wise
trained unary model as a post-processing step. These
labelling accuracy by optimizing the softmax classifica-
methods first train a CNNs model or generate CNNs
tion loss for the semantic labelling task. For the depth
features for constructing the unary potential, then add
estimation problem, depths captured by depth sensors
spatial pairwise constraints to optimize the CRFs loss.
are often noisy and contain missing values due to poor
In the pioneering work of [16], Farabet et al. present a
illumination and sensor limitation, etc. We tackle this
multi-scale CNNs framework for scene labelling, which
problem by proposing a robust depth estimation model,
usesCRFsasapost-processingstepforlocalrefinement.
which uses the robust Turkey’s biweight loss.
Most recently, Chen et al. [17] propose to first train a
We summarize the main contributions of this work as
fully convolutional neural networks for pixel classifica-
follows.
tion and then separately apply a dense CRFs to refine
• We present a general deep continuous CRFs learn- the semantic labelling results. Later on, more attempts
ing model for both discrete and continuous pixel- are made towards joint learning of CNNs and graphi-
level labelling problems. Both the unary and the cal models. In the work of [5], Zheng et al. propose to
pairwise potentials are modeled as CNNs networks, implementthemeanfieldinferenceinCRFsasrecurrent
and jointly learned in an end-to-end fashion. There neuralnetworks(RNNs)tofacilitatetheend-to-endjoint
exists a closed-form solution for the MAP infer- learning. Lin et al. [18] propose a piecewise training ap-
enceproblem,largelyreducingthecomplexityofthe proach to learn the unary and pairwise CNNs potentials
model. OurCRFsmodelisfullyconnectedandthus in CRFs for semantic labelling. They further propose
may better capture long-range relations. in [19] to directly learn CNNs message estimators in the
• Instead of the commonly used maximum likelihood messagepassinginferenceratherthanlearningthepoten-
criterionforCRFsparameterlearning,weshowthat tial functions. All of the above-mentioned joint learning
a task-specific loss function can be applied to di- approaches exploit discrete CRFs, which need to design
rectly optimize the MAP estimates, and show two approximation methods for both learning and inference.
different applications, namely, multi-class seman- Incontrast,wehereexplorecontinuousCRFs,whichen-
tic labelling (discrete) and robust depth estimation joys the benefits of exact maximum likelihood learning
(continuous). and closed-form solution for the MAP inference.
• We experimentally demonstrate that the proposed
frameworkshowsadvantagesoverthebaselinemeth- CNNs for depth estimation For the depth estima-
ods and state-of-the-art methods on both semantic tion task, Eigen et al. [7] propose to train multi-scale
labelling and robust depth estimation tasks. CNNs to directly output the predicted depth maps by
optimizing the pixel-wise least square loss for depth re-
There seems to be a common understanding that for gression. During the course of network training, there
discrete labelling problems, continuous CRFs models is no explicit structured constraints involved. Most re-
may not be suitable. As a result, an increasingly sophis- cently, Liu et al. [8] propose a deep convolutional neu-
ticated and computationally expensive discrete CRFs ral fields model for depth estimation from single images,
methods, in terms of both training and inference, have which jointly learns continuous CRFs and deep CNNs
been developed in the literature. Here we argue that a in a single framework. They further propose a more
carefully-designed, yet much simpler, continuous CRFs effcient training approach based on fully convolutional
model is capable of solving discrete labelling problems ef- networks and a superpixel pooling method in [13]. Our
fectivelyandproducingcompetitiveresults, atleastinthe workhere,whichalsojointlylearncontinuousCRFsand
caseofdeepCRFs—CRFswithdeepCNNsmodelingthe CNNs in a single framework, is mainly inspired by these
potentials. two works [8,13]. However, our work differs from [8,13]
2
in the following important aspects. (a) Our method ad- Instead of the traditionally optimized log-likelihood
dresses both discrete and continuous labelling problems, objective, we propose to directly optimize the quality
while the works in [8,13] only deal with continuous la- of the estimated labelling (MAP solution). Similar to
belling problem; Apart from continuous prediction, we the discriminative max-margin framework proposed by
extend continuous deep CRFs with closed-form MAP Taskar et al. [20], our learning objective is:
inference for discrete prediction. We show that by op-
N
timizing the task-specific empirical risk, it is possible minλ(cid:107)θ(cid:107)2+(cid:88)L(yˆ(i),y(i)),
toachievecompetitiveperformancefordiscretelabelling θ 2 2
i=1
tasks. (b) In contrast to the maximum likelihood opti-
where yˆ(i) =argmaxPr(y|x(i);θ). (4)
mization in [8,13], we show the flexibility and usefulness
y
of employing a task-specific loss in the context of deep
CRFs. (c) Our method here is based on fully-connected Here L(·,·) is a task-specific loss function, which mea-
CRFs with both the unary and the pairwise potentials sures the discrepancy of the predicted label with the
modeled as deep CNNs. In contrast, the pairwise term ground-truthlabel;y(i),yˆ(i) arerespectivelytheground-
in [8,13] is a linear function (or single layer) built upon truth and the predicted labels of the i-th training exam-
hand-crafted image features. In our model, the pair- ple. The loss function L provides important information
wise term is also a CNNs model, and all the features on how good a potential prediction yˆ(i) is with respect
are learned end-to-end. to the ground-truth label y(i). This is generally not ad-
dressedinthetraditionalCRFslearningapproachesthat
rely on the maximum likelihood scheme. To our knowl-
2 Fully-connected deep continu-
edge, this has not been explored in the case of learning
ous CRFs deepCRFsparameters. Notethattheimplicitdifferenti-
ation technique can be applied to compute the gradient
of L(yˆ(i),y(i)) with respect to the parameters θ, which
Fig.1showsanoverviewofourfully-connecteddeepcon-
enables us to use stochastic gradient descent (SGD) to
tinuous CRFs model, which consists of a unary network
learn deep structured models.
and a pairwise network. The unary network generates
the unary prediction and is parametrized by θU, while
the pairwise network outputs a similarity matrix of su- 2.2 Energy formulation
perpixel pairs and is parametrized by θV. The final pre-
We now show how to generalize the deep continuous
diction is performed by combining the unary and the
CRFs model in [13] to fully-connected multi-label dis-
pairwise outputs to conduct the MAP inference.
crete labelling problems, e.g., multi-class image segmen-
tation.
2.1 Discriminative learning of model pa- For the multi-class image segmentation task, given an
rameters imagex,werepresentthelabelofeachnode(superpixel)
asanmdimensionalvectory (y ∈Rm),whichconsists
p p
Given N i.i.d. training examples D = {(x(i),y(i))}Ni=1, of the confidence scores of superpixel p being each of
CRFs typically learn the model parameters θ by the the m classes. Here m is the total number of semantic
maximum likelihood learning, i.e., minimizing the nega- classes. Let y = y (cid:12)y (cid:12)...(cid:12)y denotes the overall
1 2 n
tive conditional log-likelihood: segmentation label of the image x, with (cid:12) stacking two
vectors and n being the number of superpixels in x. We
N
minλ(cid:107)θ(cid:107)2−(cid:88)logPr(y(i)|x(i);θ), (1) formulatetheenergyfunctionasatypicalcombinationof
θ 2 2 unary potentials and pairwise potentials over the nodes
i=1 (superpixels) N and edges S of the image x:
where λ(cid:107)θ(cid:107)2 istheregularizationterm,withλbeingthe (cid:88) (cid:88)
2 2 E(y,x)= U(y ,x)+ V(y ,y ,x), (5)
weight decay parameter. The conditional log-likelihood p p q
is modelled as: p∈N (p,q)∈S
whereU,V aretheunaryandpairwisepotentialfunction
1
Pr(y|x)= exp(−E(y,x)), (2) respectively.
Z(x)
The unary potential is formulated as:
where E is an energy function as described in Sec. 2.2;
U(y ,x;θU)=(cid:107)y −z (cid:107)2. (6)
Z is the partition function defined as: Z(x) = p p p 2
(cid:82)
exp{−E(y,x)}dy for continuous labelling y. During Here z is the unary network (network architecture de-
y p
prediction, one performs the MAP inference: tailed in Fig. 2) output for the superpixel p, which is
parametrized by θU; (cid:107)·(cid:107)2 denotes the squared (cid:96) norm
yˆ =argmaxPr(y|x;θ). 2 2
(3) of a vector.
y
3
Unary prediction
Unary network
Pairwise network Final prediction
Pairwise similarity
Figure1: Thepredictionframeworkofourmethodforthesemanticlabellingtask. Theimageisfirstover-segmentedintosuperpixels,
andthenasinputtotheunaryandthepairwisenetworks. Theunarynetworkoutputstheunarypredictionofeachsuperpixel,andthe
pairwisenetworkoutputsthepairwisesimilaritiesofsuperpixepairs. Thefinalpredictionisobtainedbycombiningtheoutputsofthe
unaryandthepairwisenetworkstosolvetheMAPinference(Eq. (12)). Detailsoftheunaryandthepairwisenetworksareillustrated
inFig. 2.
Conv block 1: Conv block 2: Conv block 3: Conv block 4: Conv block 5: Conv block 6: FC layers:

3×3 conv 64 3×3 conv 128 3×3 conv 256 3×3 conv 512 3×3 conv 512 7×7 conv 4096 fc 512
3×3 conv 64 3×3 conv 128 3×3 conv 256 3×3 conv 512 3×3 conv 512 3×3 conv 512 fc 128
2×2 pooling 2×2 pooling 3×3 conv 256 3×3 conv 512 3×3 conv 512 3×3 conv 512 fc m
2×2 pooling 2×2 pooling 2×2 pooling sp pooling
Conv block 1: Conv block 2: FC layer: Gaussian kernel layer:

3×3 conv 64 3×3 conv 128 fc 128 calc R in Eq. 8
3×3 conv 64 3×3 conv 128
2×2 pooling 3×3 conv 128
sp pooling
Figure 2: Theunary(upperplot)andthepairwise(lowerplot)networkarchitecturesofourmethod. Theunarynetworkiscomposed
of6convolutionblocksand3fully-connectedlayers. Thepairwisenetworkisrelativelyshallow,whichconsistsof2convolutionblocks
togetherwithonefully-connectedlayerandaGaussiankernellayertocalculatethepairwisesimilaritymatrixRinEq. (8). misthe
numberofsemanticclasses. Weusethesuperpixelpooling(denotedassppoolinglayer)methodasproposedin[13]toobtainsuperpixel
CNNsfeaturesfromtheconvolutionmaps.
The pairwise potential is written as: With Eqs. (6) and (7), the energy function in Eq. (5)
can now be written as:
1
V(y ,y ,x;θV)= R (cid:107)y −y (cid:107)2. (7)
p q 2 pq p q 2 E(y,x)= (cid:88)(cid:107)y −z (cid:107)2+ (cid:88) 1R (cid:107)y −y (cid:107)2
p p 2 2 pq p q 2
HereRpq,parameterizedbythepairwisenetworkparam- p∈N (p,q)∈S
eters θV, is the output of the pairwise network (network =y(cid:62)Ay−2z(cid:62)y+z(cid:62)z, (9)
architecture detailed in Fig. 2) from the superpixel pair
(p,q). Specifically, Rpq is computed from a Gaussian where
kernel layer:1
A=A ⊗I , (10)
R (θV)=βexp(−(cid:107)s −s (cid:107)2−γ(cid:107)l −l (cid:107)2), (8) 0 m
pq p q 2 p q 2 with A =I +D−R. (11)
0 n
wheres ,s aretheconvolutionfeatures(withdimension
p q
being 128 as illustrated in Fig. 2) for superpixel p, q Herez=z1(cid:12)z2(cid:12)...(cid:12)zn;Im,In areidentitymatricesof
respectively; l , l are the centroid coordinates of the size m×m, n×n respectively; ⊗ denotes the Kronecker
p q
superpixelpandqrespectively(normalizedto[0,1]);γ is product. R is the n×n matrix composed of Rpq; D is a
(cid:80)
ahyperparameterthatcontrolsthescaleoftheGaussian diagonal matrix with Dpp = qRpq.
position kernel; β is a non-negative scaling factor, which
is regarded as one of the pairwise network parameters
2.3 Optimization
and is jointly learned in our framework.
1NotethatwearenotlimitedtotheGaussiankernelhere. We MAP Inference To predict the label of a new given
canuseothersimilaritycalculationformssuchasLaplacian. image x, we perform the maximum a posterior (MAP)
4
inference, which is: categories. In this circumstance, we minimize the soft-
maxloss,typicallyemployedinmulti-classclassification,
yˆ =argmaxPr(y|x)=argminE(y,x). (12) i.e., maximize the label-wise classification accuracy, to
y y
directly pursue a desirable global accuracy:
With the energy formulation in Eq. (9), we can obtain
the closed-form solution for Eq. (12):
m
L(yˆ,y)=−(cid:88)(cid:88)y log exp(−yˆpj) . (15)
yˆ =A−1z, (13) p∈Nj=1 pj (cid:80)mk=1exp(−yˆpk)
where A−1 = (A0⊗Im)−1 = A−01⊗Im, with A−01 can Then after we get the prediction yˆp, the predicted cate-
be obtained by solving a linear equation system. Note gory label for the p-th superpixel is:
that for computational efficiency, one typically does not
need to compute the matrix inverse explicitly. tˆ=argmaxyˆ . (16)
pj
Discussion on MAP inference of fully connected CRFs j
Note that even for fully connected models, the MAP in-
In the experiment section, we show that the use of this
ference here can still be calculated efficiently, due to the
classification loss, instead of the traditional maximum
closed form of Eq. (13). In contrast in the case of dis-
likelihood loss, is critically important in achieving good
crete fully-connected CRFs models, efficient inference,
performanceformulti-classlabellingtaskswithourdeep
even with various approximations, is only possible for a
continuously-valued CRFs model.
very special case to date. As indicated in [21], in or-
der to apply the approximated fast inference of [21], the
Robust depth regression For the depth estimation
pairwise term must be in the form of the Gaussian ker-
problem, one of the main criteria for a good predictions
nel, and the input features must be in low dimension,
is the root mean square (rms) error. Therefore, we can
typically less than 20. In our case, the input feature’s
train our model by directly pursuing good MAP esti-
dimension is 128, which is much higher than what can
mates in terms of RMS measure. To achieve this, the
be possibly handled by [21].
loss function for the depth estimation problem can be
written as:
2.3.1 Maximum likelihood learning
(cid:88)
L(yˆ,y)= ρ(r ), (17)
Since here we consider the continuous domain of y (vec- p
tor of confidence scores), the integral in the partition p∈N
function in Eq. (2) can be analytically calculated un-
wherer =y −yˆ isthedepthresidualofthesuperpixel
p p p
der certain circumstances, as shown in [13]. Accord-
p regarding to the ground truth; ρ(·) is a penalty func-
ing to [13], the negative conditional log-likelihood can
tion, which generally satisfies ρ(0)=0 and ρ(r )>0 for
p
be written as:
r (cid:54)=0.
p
Astraightforwardalternativetothelog-likelihoodloss
−logPr(y|x)=y(cid:62)Ay−2z(cid:62)y+z(cid:62)A−1z
is the least squares (LS) loss:
1 n
− log(|A|)+ log(π). (14)
2 2 ρ(r )=r2, (18)
p p
The gradients of −logPr(y|x) with respect to A and z
However, LS loss is generally not robust to outliers. Ac-
canbeanalyticallycalculatedasshownin[13]. Thenap-
tually all convex regression loss functions are considered
plying the chain rule (back propagation), the gradients
notrobust. Forexample,forthoseextremelylargeresid-
with respect to θU, θV and β can be calculated accord-
uals (typically outliers), the LS loss imposes too large
ingly.
penalty, which inevitably biases the model towards out-
liers.
2.3.2 Task specific learning To fully exploit the flexibility of the proposed task-
specific CRF parameter learning, which allows us to
Multi-classsemanticlabelling Inthemulti-classse-
choose arbitrary learning criterion, here we consider a
mantic labelling task, we obtain a prediction of m di-
non-convex, robust regression loss function, namely, the
mensional vector yˆ for superpixel p, with each yˆ (j-
p pj
Turkey’s biweight loss:
th element of yˆ , j = 1,2,...,m) indicating the con-
p
fidence score of the superpixel p belonging to the j-th
category. For the training ground truth yp, the confi- ρ(r )=(cid:40) c62(cid:2)1−(1− rcp22)3(cid:3), if|rp|<c (19)
p
dence value for the ground truth class is 1, and 0 for c2, otherwise.
6
other classes. During optimization, we prefer the confi-
dence score of the ground-truth category being as large Here c is a pre-defined constant. Fig. 3 shows the
aspossiblecomparedtotheconfidencescoresofincorrect Turkey’s biweight loss and its first-order derivative.
5
Ground Structure Furniture Props Pix.Acc. ClassAcc.
Guptaetal.[22] - - - - 78.0 -
Couprieetal.[23](RGB+D) 87.3 86.1 45.3 35.5 64.5 63.5
Khanetal.[24] 87.1 88.2 54.7 32.6 69.2 65.6
Mulleretal.[25] 94.9 78.9 71.1 42.7 72.3 71.9
Eigenetal.(RGB)[6] - - - - 75.3 -
Eigenetal.(RGB+D+N)[6] 93.9 87.9 79.7 55.1 80.6 79.1
Ours(unary) 84.3 84.8 77.3 54.6 77.1 75.8
Ours(full,RGB) 88.0 89.8 77.7 58.0 80.8 78.7
Ours(full,RGB+D) 92.9 88.7 81.7 61.7 82.5 81.2
Table 1: Segmentation results on the NYU v2 dataset (4-class). Note that our full model even outperforms the RGB+D+N model
in[6]whichusesadditionalground-truthdepthandsurfacenormalinformation.
wall floor cabinet bed chair sofa table door window bookshelf picture counter blinds desk shelves curtain dresser pillow mirror floormat
Guptaetal.[26] 68.0 81.3 44.9 65.0 47.9 47.9 29.9 20.3 32.6 18.1 40.3 51.3 42.0 11.3 3.5 29.1 34.8 34.4 16.4 28.0
Eigenetal.[6] 68.2 83.3 45.0 51.9 46.0 40.4 32.0 17.6 33.0 31.5 45.0 53.4 43.9 11.3 8.9 35.4 26.3 31.0 32.6 27.4
Ours 67.7 75.9 46.8 52.2 43.8 46.3 29.0 27.3 33.2 35.9 48.6 41.9 46.6 9.0 7.8 32.9 28.9 24.6 18.1 23.0
clothes ceiling books fridge tv paper towel s-curtain box w-board person n-stand toilet sink lamp bathtub bag other-struc other-fur other-prop
Guptaetal.[26] 4.7 60.5 6.4 14.5 31.0 14.3 16.3 4.2 2.1 14.2 0.2 27.2 55.1 37.5 34.8 38.2 0.2 7.1 6.1 23.1
Eigenetal.[6] 12.9 75.2 16.6 15.7 32.3 20.3 14.8 21.7 4.5 6.6 27.5 24.2 44.8 39.4 33.6 29.1 1.5 11.7 6.8 29.4
Ours 15.6 52.6 19.3 22.3 40.7 17.2 22.5 24.1 5.6 46.9 47.6 19.9 50.6 37.8 24.1 10.3 4.2 13.4 10.4 26.3
Table 3: Theper-classpixel-wiseJaccardindexforeachofthe40categoriesontheNYUv2dataset. Ourmethodperformsthebest
on18outofthe40classes. NotethatEigenet al.[6]useextrasurfacenormalinformation,andtheirnetworktakesmulti-scaleinput
images.
Error Accuracy
Noiselevel Method (lowerisbetter) (higherisbetter)
rel log10 rms δ<1.25 δ<1.252 δ<1.253
Logloss 0.223 0.094 0.835 0.611 0.891 0.973
0%
Robustloss 0.219 0.091 0.787 0.632 0.894 0.974
Logloss 0.263 0.115 1.023 0.518 0.815 0.945
10%noise
Robustloss 0.222 0.092 0.790 0.628 0.896 0.975
Logloss 0.369 0.130 1.256 0.461 0.764 0.915
25%noise
Robustloss 0.218 0.093 0.815 0.618 0.894 0.976
Logloss 0.786 0.211 2.152 0.246 0.477 0.685
10%outlier
Robustloss 0.221 0.092 0.812 0.617 0.895 0.974
Logloss 0.951 0.261 2.503 0.150 0.371 0.617
25%outlier
Robustloss 0.235 00.094 0.819 0.611 0.888 0.972
Table 4: Comparisonsofdepthestimationperformanceofdifferentlossfunctionswithrespecttodifferentnoise/outlierratiosonthe
NYUv2dataset.
6
Testimage Ground-truth Ourpredictions
Figure 5: Examples of depth predictions on the KITTI dataset (Best viewed on screen). Depths are shown in log scale and in color
(redisfar,blueisclose).
100 80
Softmax−loss Softmax−loss
Log−loss 70 Log−loss
Loss00..0001..5512 Gradient−−00000.....012321 Pixel−wise accuracy 24680000 Per−category accuracy123456000000
−0.3
−02 −1 0 1 2 −0.−42 −1 0 1 2 0 4 13Number of classes 40 0 4 13Number of classes 40
Residual Residual
Figure 3: An illustration of the Turkey’s biweight loss and its Figure 6: Performance comparison of the conditional log-
firstorderderivative. likelihoodlossandthesoftmaxlossforimagesegmentationonthe
NYUv2dataset.
Pix.Acc. ClassAcc. Avg.Jaccard Freq.Jaccard
Guptaetal.[22] 59.1 28.4 27.4 45.6
Guptaetal.[26] 60.3 35.1 28.6 47.0
Khanetal.[24] 50.7 43.9 - 42.1
Longetal.[4](RGB) 60.0 42.2 29.2 43.9
Longetal.[4](RGB+D) 61.5 42.4 30.5 45.5
Eigenetal.[6](RGB+D+N) 62.9 41.3 30.8 47.6
Ours(unary) 58.8 38.2 27.1 44.6
Ours(full,RGB) 62.6 42.2 30.5 49.1
Ours(full,RGB+D) 63.1 39.0 29.5 48.4
Table2: State-of-the-artcomparisonontheNYUv2dataset(40-
class).
3 Experiments
Todemonstratetheeffectivenessoftheproposedmethod,
we perform experiments on the tasks of multi-class se-
manticlabellingandrobustdepthestimationfromsingle
images. Thefirstoneisdiscrete-valuedlabellingandthe
second one is continuously-valued labelling.
Implementation details We implement our method
based on the popular CNNs toolbox MatConvNet2. The
Testimage Ground-truth Unaryprediction Ourfullmodel network training is performed on a standard desktop
withanNVIDIAGTX780GPU.Wesetthemomentum
Figure 4: Semantic labelling (4-class) examples on the NYU v2
as 0.9 and the weight decay parameter as λ = 0.0005.
dataset. Comparedtotheunarypredictions,ourfullmodelyields
moreaccurateandsmootherlabelling. The first 5 convolution blocks and the first layer of the
2http://www.vlfeat.org/matconvnet/
7
0.7 Robust loss 0.7 Robust loss are concatenated after the superpixel pooling layer. The
0.65 Log−loss 0.6 Log−loss
compared results together with the per-class accuracy
δAccuracy ( <1.25)00..0045..5556 δAccuracy ( <1.25)0000....2345 oTdneapbttlhheec1Nu.eYAsUsouwvt2pee4cra-fconlramssesses,setogaumtree-fonuftl-altthmieoo-nadrettalsmwkietathhreoindrsceopbroyproatrecadotneind-
0.4 0.1 siderablemargin. Itisworthmentioningthatourmethod
0.350 10 20 30 40 50 00 10 20 30 40 50 outperformsEigenet al.[6],whichusedRGBimagesto-
Noise percentage Outlier percentage
getherwithdepthsandsurfacenormals. Comparedwith
theunary-onlymodel,ourfullmodelwhichjointlylearns
Figure 7: Predictionaccuraciesofthelog-likelihoodlossandthe
robustregressionlossfordepthestimationwithrespecttodifferent the unary and the pairwise networks shows consistent
percentagesofnoises/outliersontheNYUv2dataset. improvements in terms of all the metrics, demonstrating
that the continuous CRFs model indeed improves dis-
Error Accuracy crete labelling. Fig. 8 shows some qualitative examples
Method (lowerisbetter) (higherisbetter)
rel log10 rms δ<1.25 δ<1.252 δ<1.253 of our method together with the unary baseline. Com-
Saxenaetal.[27] 0.280 - 8.734 0.601 0.820 0.926 pare to the unary predictions, our method yields more
Eigenetal.[7] 0.190 - 7.156 0.692 0.899 0.967
accurate and smoother labellings.
Liuetal.[13] 0.217 0.092 7.046 0.656 0.881 0.958
Ours 0.203 0.086 6.427 0.684 0.894 0.965 Table2summarizestheresultsonthe40-classsegmen-
tation task. Our method generally achieves better or
Table 5: Comparisons of depth estimation performance on the
comparable performance against state-of-the-art meth-
KITTIdataset. Theresultsof[7]areobtainedbyusingmillionsof
trainingimageswhileoursareobtainedbytrainingon700images. ods [4,6]. Specifically, with RGB only as the input, our
methodoutperforms[4]usingsimilarsettings. Notethat
in [4], Long et al. achieved better results by using HHA
6th convolution block of the unary network in Fig. 2 are
encoding,whichrevealsthatsimilarstrategiescanbeap-
initialized from the VGG-16 [28] model. All layers are
plied here to further improve our results. In [6], Eigen
trained using back-propagation. The learning rate for
et al. have used RGB images together with the ground-
randomly initialized layers is set to 10−5; for VGG-16
truth depths and surface normals as the input to their
initialized layers it is set to a smaller value 10−6. The
network training. Yet, our method achieves compara-
hyperparameter of the position Gaussian kernel is set to
bleperformance. Withdepthchannelsincorporated,our
γ = 0.1. The parameter c in the Turkey’s biweight loss
method obtains the best global labelling accuracy. Our
(Eq.(19))issetto1. WeuseSLIC[29]togenerate∼700
fullmodelagainshowsimprovementsovertheunaryonly
superpixels for each image.
model, demonstrating the usefulness of the joint learn-
ing scheme. The per-class pixel-wise Jaccard indexes of
3.1 Multi-class semantic labelling the 40 object categories are presented in Table 3, with
our method achieves the best results on 18 out of the 40
The semantic labelling experiments are evaluated on the classes.
NYU v2 [30] and the MSRC-21 [31] datasets. The NYU MSRC-21 Table 6 reports the segmentation results
v2datasetconsistsof795imagesfortrainingand654im- ontheMSRC-21dataset. Aswecansee,ourresultsper-
agesfortest(weusethestandardtraining/testsplitpro- forms comparable to state-of-the-art methods [32], [33].
vided with the dataset). For the semantic label ground-
truth, we use the 4 and 40 classes described in [30], [22] Pix.Acc. ClassAcc.
respectively. The performance are evaluated using the Lucchietal.[34] 83.7 78.9
Royetal.[32] 91.5∗ -
commonly applied metrics as in [6,23,24,26], namely,
Liuetal.[33] 88.5 86.7
pixel-wiseaccuracy(Pix.Acc.) andaverageper-classac- Liuetal.[33](co-occur) 91.1∗ 90.5∗
curacy (Class Acc.). For the 40-class segmentation task, Ours(softmaxloss,unary) 89.6 86.4
we also report the average Jaccard index (Avg. Jaccard) Ours(softmaxloss,full) 91.4 89.0
and the average pixel-frequency weighted Jaccard index
Table 6: Segmentation results on the MSRC-21 dataset. Note
(Freq. Jaccard) for each class. The MSRC-21 dataset that the results marked with ∗ are achieved by using additional
is a popular multi-class segmentation benchmark with mutex/co-occurrenceinformationthatmodelsrepellingrelations.
591 images containing objects from 21 categories. We
followthestandardsplittodividethedatasetintotrain-
ing/validation/test subsets. Ablation Study To show the benefits of the multi-
NYUv2Sinceground-truthdepthmapsareprovided class classification loss over the tradition log-likelihood
along with this dataset, we exploit depth cues as an ad- CRFstraining,weperformablationstudyontheperfor-
ditional setting (denoted as RGB+D). Specifically, we mancecomparisonsofthetwolossfunctionswithrespect
replicate the depth channel into 3 duplicates and regard todifferentnumbersofclasses. Inadditiontothe4-class
them as images to input into our framework. The CNNs and the 40-class segmentation tasks, we further perform
features from the RGB channels and the depth channels experimentsona13-classsegmentationtaskbymapping
8
the40categoriesinto12mainclassesanda‘background’ al. [7]. As for the training set, we use the same 700 im-
class. Here the network structures are identical and the ages that Eigen et al. [7] have used to train the method
only difference is the loss function. Fig. 6 shows the of[27]. Thecomparedresults are reported inTable 5. It
comparativeperformance. Wereportbothpixel-wiseac- canbeseenthatourmethodoutperformsthemostrecent
curacy and per-category accuracy. As illustrated in the work of [13] which have used the same training set, and
figure, the multi-class soft-max loss considerably outper- very similar CNNs structure as ours. Compared to [7],
forms the log-likelihood loss. The discrepancy becomes which have used millions of training images, our results
more significant when the number of classes increases. are very competitive with the lowest rms error achieved.
The previously-reported inferior performance of contin- Fig. 5 shows some prediction examples of our method.
uous CRFs on discrete labelling may be attributed to On both NYU v2 and KITTI datasets, our method us-
the use of the traditional maximum likelihood parame- ing robust loss outperforms the log-likelihood loss, even
ter learning. with no noises deliberately added, because (a) original
datamaycontainsomenoisesmoreorless; (b)moreim-
portantlyourmethoddirectlyoptimizesthetask-specific
3.2 Robust depth estimation
empirical risk of interest.
Weperformrobustdepthestimationontwodatasets: the WeshowsomedepthpredictionexamplesontheNYU
indoorNYUv2[30]andtheoutdoorKITTI[35]datasets. v2 dataset in Fig. 8. As we can see, our predictions well
Several commonly used evaluation metrics, i.e., relative preserve depth discontinuities and align to local details.
error (rel), log10 error (log10), root mean square error
(rms) and accuracy with different thresholds δ (details
4 Conclusion
arereferredto[8,13]),areusedheretoreporttheresults.
We have proposed a fully-connected deep continuous
NYU v2 The NYU v2 dataset [36] consists of 1449
CRFs learning method for both discrete and continu-
RGB-D indoor image pairs, with 795 are used for train-
ouspixel-levellabellingproblems. Theproposedmethod
ing and 654 are for testing. We have used the standard
inherits the advantages of continuous CRFs, i.e., exact
training/test split provided along with the dataset.
maximum likelihood learning and closed-form solution
In addition to performance comparisons on the origi-
for the MAP inference. Specifically, it models the unary
nal dataset, we also add additive white Gaussian noises
andthepairwisepotentialsoftheCRFsastwoCNNsand
or outliers to the ground-truth depths. For the noise
learns the network parameters in an end-to-end fashion.
case, we add additive white Gaussian noise with σ =0.1
Fordifferenttasks,wefurtherproposetooptimizeatask-
to the [0,1] normalized ground-truth depths. A certain
specific loss function, rather than the traditional log-
percentage of pixels (e.g., 10%, and 25% as shown in
likelihoodlossinlearningCRFsparameters,anddemon-
Table 4) are uniformly sampled to be added the noise.
stratetwoapplications,whicharesemanticlabellingand
Likewise, for the outliers, we simply add a large value
robustdepthestimation. Experimentalresultsshowthat
to the ground-truth of a certain percentage of uniformly
(a)theproposedmethodimprovestheunary-onlymodel
sampled pixels. Table 4 reports the compared results of
(standardregressionwithoutconsideringstructuredcon-
the log-likelihood loss (denoted as ‘log loss’) and the ro-
straints),indicatingtheeffectivenessofthepairwiseterm
bustregressionlosswithrespecttodifferentnoise/outlier
of our model; (b) our method achieves better or on par
levels. As we can observe, the robust regression loss is
results compared with state-of-the-art, even on discrete
insensitive to both noises and outliers. In contrast, the
labellingtasks,showingthepromiseoftheproposeddeep
performance of the log-likelihood loss degrades severely
structured model. We thus advocate the use deep con-
when the noise/outlier level increases. Fig. 7 illustrates
tinuous CRFs in more structured prediction problems in
the compared prediction accuracies of the two loss func-
vision, due to its power and in particular, significantly
tions. Thehorizontalaxisshowsdifferentnoise(leftplot)
simpler and efficient inference, compared with most dis-
or outlier (right plot) percentages, and the vertical axis
crete deep CRFs such as [18,19].
shows the depth prediction accuracy with δ <1.25.
References
KITTI The KITTI dataset [35] consists of outdoor
videos taken from a driving vehicle equipped with a Li-
[1] A.Coates,A.Y.Ng,andH.Lee, “Ananalysisofsingle-
DaR sensor. The ground-truth depths of the KITTI
layer networks in unsupervised feature learning,” in
dataset are scattered at irregularly spaced points, which
Proc. Int. Conf. Artificial Intell. & Stat., 2011.
only consists of ∼5% pixels of each image, with all oth-
[2] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Ima-
ers unlabelled. We here treat those unlabelled regions
geNet classification with deep convolutional neural net-
as outliers and train the network using robust regression works,” in Proc. Adv. Neural Inf. Process. Syst., 2012.
loss. Theresultsarereportedbyusingthesametestset, [3] J. Tompson, A. Jain, Y. LeCun, and C. Bregler, “Joint
i.e., 697 images from 28 scenes, as provided by Eigen et training of a convolutional network and a graphical
9
Testimage Ground-truth Ourprediction
Figure 8: DepthpredictionexamplesofourmethodontheNYUv2dataset.
10