Recovering Human Body Configurations- Combining Segmentation and Recognition

8 Pages·1.519 MB·English

by  Mori G., Ren X., Efros A.

Recovering Human Body Configurations: Combining Segmentation and
Recognition
GregMori,XiaofengRen,AlexeiA.Efros† andJitendraMalik
ComputerScienceDivision †RoboticsResearchGroup
UCBerkeley UniversityofOxford
Berkeley,CA94720,USA OxfordOX13PJ,U.K.
rately find people in images such as these would facilitate
manyusefulapplicationssuchasinitializing3Dkinematic
trackers,understandinghumanactions,andre-renderingfor
graphics.
The difficulties described above have led researchers to
simplifytheproblem,oftenusingdatasetsofunclothedpeo-
ple, or those processed with background subtraction from
video sequences. The range of variation in pose is usually
(a) (b) (c) limited,andthereislittleornobackgroundclutter.
Figure1:Theproblem:(a)Inputimage.(b)Extractedskeletonof Inthisworkweattempttotacklethisprobleminamore
localizedjointsandlimbs.(c)Segmentationmaskassociatedwith generalsetting. Thedatasetweuseisacollectionofsports
humanfigure. newsphotographsofbaseballplayerscollectedfromtheIn-
ternet. Theimagesselectedarefullbodypicturesofasin-
Abstract
gle player. These players are in a wide variety of poses,
arewearingdifferentclothesthatareoftentextured,andare
Thegoalofthisworkistotakeanimagesuchastheonein
photographed outdoors under varying lighting conditions,
Figure 1(a), detect a human figure, and localize his joints
innatural,clutteredscenes.
and limbs (b) along with their associated pixel masks (c).
The structure of this paper is as follows. In Section 2
Inthisworkweattempttotacklethisprobleminageneral
wediscussrelatedpreviouswork. Insection3wedescribe
setting. The dataset we use is a collection of sports news
our approach at a high level. Section 4 and 5 provides the
photographs of baseball players, varying dramatically in
detailsofouralgorithm.Wediscussthepositiveaspectsand
poseandclothing. Theapproachthatwetakeistouseseg-
limitationsofourmethodinsection6.
mentationtoguideourrecognitionalgorithmtosalientbits
of the image. We use this segmentation approach to build
2. Related Work
limb and torso detectors, the outputs of which are assem-
bledintohumanfigures. Wepresentquantitativeresultson
torso localization, in addition to shortlisted full body con- Someoftheearliestresearchrelatedtothisproblemisthe
figurations. pedestrian tracking work of Hogg [3]. A vast quantity of
workcontinuedinthisvein, usinghighdegree-of-freedom
1. Introduction 3D models of people, rendering them in the image plane,
andcomparingthemwithimagedata. Gavrila[2]provides
The goal of this work is to take an image such as the a survey of this work. These approaches typically require
one in Figure 1(a), detect a human figure, and localize his a hand-initialized first frame, and the large number of pa-
jointsandlimbs(b)alongwiththeirassociatedpixelmasks rametersintheirmodelsleadtodifficulttrackingproblems
(c). This problem is arguably the most difficult recogni- in high dimensional spaces. More recent developments in
tion problem in computer vision. Difficulties arising from pedestriandetection, suchasMohanetal.[7]andViolaet
appearance variation due to clothing are compounded by al.[18],arefairlysuccessfulindetectingpeopleincommon
articulated deformation. Consider the people in Figure 2. standing poses. However, these template-based window-
Their poses are not probable, yet are definitely possible. scanning approaches do not localize joint positions, and it
Moreover, the appearance of their limbs varies rather dra- isnotclearwhethertheygeneralizetofindingpeopleinar-
maticallybetweenthedifferentpeople. Theabilitytoaccu- bitraryposes.
1
Recently, there has appeared promising work exploring
theinterplaybetweenlow-levelcuesandhigh-levelknowl-
edge. ThelineofresearchbyTorralbaetal.[15]develops
relationshipsbetweenlocalobjectdetectorsandtheglobal
scene context. Others combine segmentation with recog-
nition, making more sophisticated uses of low-level cues.
Yu and Shi [19] add object-specific patch-based templates
into the Normalized Cuts framework. Tu et al. [17] com-
Figure2:Thechallengeofunlikelypose
binefaceandletterdetectorswithsegmentationintheDD-
MCMCframework.However,mostoftheseapproachesare
The complexities in 3D model-based tracking have led
testedonsimplerigidobjectsanditisyettobeshownhow
researcherstoposetheproblemasoneofmatchingtostored
theseframeworkshandlecomplicatedobjectssuchasartic-
2Dexemplars. ToyamaandBlake[16]usedexemplarsfor
ulatedhumanbodiesinarbitraryposes.
trackingpeopleas2Dedgemaps. MoriandMalik[8],and
SullivanandCarlsson[14]directlyaddresstheproblemwe 3. Motivation and Approach
are interested in. They stored sets of 2D exemplars upon
which joint locations have been marked. Joint locations One classic approach to recognizing objects such as peo-
aretransferredtonovelimagesusingshapematching. The ple would be to model them as a collection of generalized
problemwithsuchexemplarbasedapproachesisthecom- cylinderswhichonecouldaimtodetectinabottom-upap-
binatorial explosion in the number of exemplars needed to proach (“ribbon finding”). These strategies were common
matchawidevarietyofposes. Shakhnarovichetal.[12]at- in the 1980s, but have fallen into disfavor in recent years.
tempt to address variation in pose and appearance through Thebasicproblemisthattryingtoreliablydetecteachpart
brute force, using a variation of locality sensitive hashing individuallyisproblematic(ifnothopeless)inareal-world
for speed to match upper body configurations of standing, setting.Zoominonanimageofaperson’sarmandyouwill
frontfacingpeople.Itisnotclearwhethersuchanapproach findthatitlooksnodifferentthanapieceofgrass,oratree
willscaletohandleavarietyofposessuchasthoseinFig- trunk. Butlookatthatsamearmwithinitscontext(ahand,
ure 2. One fundamental problem with the exemplar-based ashoulder,asideofatorso),anditinstantlybecomesrec-
methodsisthattheydonottakefulladvantageofthesalient ognizable. Thatis,mostlow-levelfeaturesareinformative
low-levelcues. onlywhenconsideredwithintheircontext,whichrepresents
Another family of approaches model people as an as- theglobalinformationpresentinthescene. But,ofcourse,
sembly of parts. Typically a simple low-level detector is thecontextismadeupoflow-levelfeaturesaswell,leading
applied to produce a set of candidate parts. Then a top- toaclassicchicken-and-eggproblem. Howdowecrackit?
downproceduremakesinferencesaboutthepartsandfinds Let us start with the few low-level features that are, in
the best assembly. Song et al. [13] detect corner features fact,informativeindependentofthecontext. Thesefeatures
in video sequences and model their joint statistics using usuallyrepresentpartsthataresalient,thatstandout,having
tree-structured models. Felzenswalb and Huttenlocher [1] enoughinformationinthemselves. Thisisakintoviewing
score rectangles using either a fixed clothing model or sil- a modern painting, say a Picasso canvas from his Cubist
houettes from background subtraction of video sequences period: amongacrowdofcubesafewsalientshapesjump
andthenquicklyfindanoptimalconfigurationusingthedis- out – an elbow, a hand, a face. We call them islands of
tance transform to perform dynamic programming on the saliency. Starting from a set of these islands a viewer is
canonical tree model. Ioffe and Forsyth [4] use a simple usually able to explain the rest of the picture by making
rectangledetectortofindcandidatesandassemblethemby educatedguessesbasedoncontext(“ifthat’sanelbowand
samplingbasedonkinematicconstraints. that’s a torso, then this line must be an arm”), in a sense,
The difficulty with the tree-based methods is that there bridgingtheislandsofsaliency. Weoperationalizethisidea
aredependenciesamongthebodypartsthatcannotbecap- byusinganumberoflow-levelpre-processingstages.
tured by a tree. For example, there are anthropometric The low-level procedure gives us a collection of salient
constraints (we use data from [9]) on the relative scales of parts,inourcasetorsosand“half-limbs”(upperandlower
the limbs. Symmetry in appearance between left and right legsorarms). Notethatweexpectthislow-levelprocessto
limbs, such as the arm in Figure 2(left) is another cue that benoisy–manycorrectpartswillnotappearsalientwhile
can’tbecapturedinatree. Reasoningaboutself-occlusion somesalientpartswillturnouttobewrong. Allweneedis
cannot be done either. Finally, in a tree model there is no enoughgood,salientpartstojump-startoursearch. “Partial
directmechanismforpreventingthereuseofimagepixels. configurations” consisting of a few half-limbs and a torso
Anarmwithagoodlow-levelscorecouldbelabeledasboth are assembled from these salient parts. There is a combi-
therightandleftarm. natorial problem to determine which parts can be used to-
2
(a) (b) (c) (d) (e) (f)
Figure3:Imageanalysisatthelow-level:(a)Inputimage.(b)(c)Cannyedgesattwodifferentscales.Textureonclothingandbackground
clutterposesevereproblemsforrecognition.(d)pb(“probabilityofboundary”)image[6].Ithandleshigh-frequencytexturenicely.(e)A
NormalizedCutssegmentationwithk = 40. Salientlimbspopoutassinglesegments;headandtorsoconsistofseveralsegments. (f)A
“superpixel”mapwith200superpixels.Itcapturesallthedetails.
gether to form a partial configuration. We enforce global super-
pixels
salient
constraints on human body configuration, namely relative
limbs
scales,positionsandcolors,topruneawayimpossiblecom- input partial full
image regions configurations configurations
salient
binations. Theremainingpartialconfigurationsarethenex-
finding torsos limb
tended to full human figures by searching for the missing segmentation body parts pruning completion scoring
limbs. Full configurations are sorted using a combination Figure4:Dataflowofthealgorithm.Firstanimageissegmented
oftheirindividualpartscores. intosuperpixelsandrelativelylargeregionsorsegments. Wede-
tectsalientupperandlowerlimbsfromthesesegments. Simulta-
Astandardwaytomakeuseoflow-levelinformationis
neouslywedetectpotentialheadandtorsopositions. Bothmod-
applyingtheCannyedgedetector. Figure3(b,c)showsthe
ulesreturna“shortlist”oftoprankedcandidates. Wethencom-
Canny edges of a sample image. In textured regions, sim- binethesepartsintopartialbodyconfigurationsandpruneaway
pleedgedetectorsfirefrequently,thuslimitingtheiruseful- impossibleconfigurationsbyenforcingglobalconstraintssuchas
ness. InthisworkweusetheboundaryfinderofMartinet relativescaleandsymmetryinclothing.Inthefinalstagewecom-
al. [6] (Figure 3(d)), which combines both brightness and pletepartialconfigurationsbycombinationalsearchinthespace
texture information to reduce clutter. Furthermore, we use ofsuperpixelstorecoverfullbodyconfigurations.
the Normalized Cuts algorithm [5] to group similar pixels
intoregions. Figure3(e)showsasegmentationwith40re- that we have pixel masks assigned to limbs, which would
gions. Manysalientpartsofthebodypopoutassinglere- likelyfacilitatemoreaccuratefinalscoringmechanisms.
gions,suchasthelegsandthelowerarms. Inaddition,we
use over-segmentation, asshown in Figure 3(f), consisting 4. Finding Body Parts
of a large number of small regions or “superpixels” [10],
which has been shown to retain virtually all structures in Inthissection,wedescribeourprocedureforfindingsalient
real images. These segmentations dramatically reduce the half-limbs and torsos. First, image segmentation is used
complexityoflaterstagesofanalysis,e.g.,from400K pix- to generate candidate segments. A set of low-level cues is
elsto200superpixels. Figure4showsthedataflowofour thencomputedtoclassifythesesegments. Theprocedureis
completealgorithm. empiricallyvalidatedusinggroundtruthdata.
Wevalidateourapproachinanumberofways. First,we
4.1. FindingLimbs
use hand-labeled images to evaluate the classification per-
formance of our half-limb detector, as well as the individ- As we observed in Figure 3(e), salient “half-limbs” often
ualcuesbeingused. Second, weevaluatetheperformance popoutasasinglesegment. Hencewesearchamongthese
ofoursegmentation-basedtorsodetector,againusinghand- segments,generatedbytheNormalizedCutsalgorithmwith
labeledjointpositions,andcompareittoanexemplar-based k =40,tofindhalf-limbs. Weclassifythesesegmentswith
detector. We then show how the salient half-limbs help thesetofcuesdefinedbelow.
constrain the torso position. Finally, we present results on
recovering full body configurations from the salient parts.
4.1.1 Cuesforhalf-limbdetection
Thefinalscoringoffullbodyconfigurationsisachalleng-
ing problem of its own and is not dealt with in this work. Thesetofcuesweuseforhalf-limbdetectionare: contour,
Instead, we return a shortlist of candidate body configura- shape,shading,andfocus.
tions which should be evaluated by a separate procedure. ContourCue:Theboundarycontourcuemeasureshow
Anaddedadvantageofoursegmentation-basedapproachis salientthecontourofasegmentisandhowwell-separatedit
3
isfromthebackground.Weusetheprobabilityofboundary, 4.1.2 EvaluatingtheCues
pb [6], to measure the goodness of a segment at the low-
Figure 6 shows the average number of detections for indi-
level. Thecontourcueofasegmentistheaveragepbalong
vidualcuesandforthecombinedclassifier. Wefindthatthe
itsboundary.
contourcue, orthepresenceofastrongboundarycontour,
ShapeCue: Weuseasimplifiedmodel, namelyarect- is likely the most useful. The shading cue by itself is also
angle, to capture the basic shape characteristics of half- good. The shape cue is relatively poor, showing that the
limbs.Forasegment,weestimateitssizeanditsorientation rectangle model is not sufficient to capture the shape vari-
andconstructarectangletemplatewiththeestimates. The ationsofhalf-limbs,especiallytheupper-limbs. Thefocus
shape cue is then the area of overlap between the segment cue,beingcrudeandgeneric,istheworst.Nevertheless,the
and this reconstructed rectangle. We experimented with fourcuestogethermakeagoodhalf-limbdetector.
an exemplar-based approach, using Hausdorff distance to Another way to evaluate the performance of our classi-
matchsegmentsto5exemplarhalf-limbs.Theperformance fieristolookatthepercentageofimageswhichhaveatleast
iscomparabletothesimplerectanglemodelsothatiswhat k half-limbsbeingdetected. TheresultsareshowninFig-
wasusedforourfinalresults. ure7. Inourdataset89%oftheimageshaveatleastk =3
correct half-limbs among the top 8 candidates returned by
Shading Cue: Because limbs are roughly cylindrical,
thetraineddetector.Thismotivatesustodoacombinatorial
shadingisoftenasalientcue,providingastrongsenseof3D
search for a triple of half-limbs to be combined with head
pop-out (e.g. thighs on Figure 1(a)). As can be seen from
andtorso.
Figure5,theshadingeffectispresentinmanyofthelimbs,
though in a weak and noisy form, unsuitable for methods
4.1.3 ExtendingtoFullLimbs
likeShape-from-Shading. Instead,wecapturethiseffectby
Givenahalf-limbwecaninstantiateafull-limbdetectorby
learningacoarseshadingdescriptorfromthehand-labeled
attempting to extend it in all possible directions by adding
limbs, and use it to score the potential limb regions. The
a second rectangle at a joint of the first. Note that this ex-
shadingdescriptorisconstructedasfollows: eachsegment
tension is only done to all nearby superpixels, thereby re-
is first rotated and scaled into a standard 40x100 template
ducingthecomplexityofthesearch. Theprocessofscoring
image I using the major/minor axis of the best-fitting el-
thesecondrectangleisdoneusingasegmentationcriterion
lipse. FromI,gradientimagesI andI arecomputedand
x y
similartothatof[10].
half-waverectifiedintofournon-negativechannelsI+,I−,
x x P Wext
Icyh+a,nIny−elssoatrheatthIexn=bluIrx+re−d wIx−ithanadGIayu=ssiIay+n −andIy−st.aTckheedfotuor- S(R) = Pi∈R,j∈RcWiinjt
i∈R,j∈R ij
getherintoasinglevector. Todescribetheshadingpattern Wext = e−χ2(Ci,Cj)/σc+B(i,j)/σb
on a prototypical limb, we compute a mean of all shading ij
descriptors in the training set (Figure 5). This mean shad- Wint = e−χ2(Ci,Cj)/σc
ij
ingdescriptoristhencomparedwitheachnewregionusing
whereB(i,j)istheaveragepbalongtheboundarybetween
normalizedcorrelationtoyieldashadingscore.
superpixels i and j. Essentially, this criterion is optimized
Focus Cue: The background sometimes lacks texture
when the internal similarity of a region is high, while the
orisoutoffocus. Thelackofhigh-frequencyinformation
affinitytoneighboringsuperpixelsislow.
(orenergy)isusuallyanindicationthattheregionisnotof
Thisextensionsearchcanbemademoreaccurateifwe
interest. Fortheimagesofbaseballplayersweworkwith,
know the body part label (upper or lower leg, arm, etc.),
this“focusness”isoftenausefulcue.Wemeasurethefocus
since we can be provided with constraints on the size and
C byhigh-frequencyenergyE normalizedbylow-
focus high appearanceoftheotherhalfofthelimb.Section5addresses
frequency energy E , in the following form: C =
low focus theconstrainingofthissearch.
E /((E **a)+b). The energy E and E are
high low high low
routinely estimated by convolving the image with odd and 4.2. FindingTorsos
evenfiltersandcombiningtheirresponses. Theparameters
Dominatinginsizeandrelativelyrigid,thehead/torsocom-
aandbareselectedbycross-validation, withroughlya =
bination has a characteristic shape. Connecting the other
0.3andb=0.
partstogether,thetorsoisalsothemostcriticalinrecover-
CombiningtheCues: Thecueswehavedefinedabove ingthebodyconfiguration. Tofindthetorsos,weagainuse
are not directly comparable to one another. We use a sig- a segmentation-based approach. We test its performance
moidfunctiontotransformeachcueintoaprobability-like against an alternative approach based on exemplars. The
quantity and then combine them linearly. The weights are metric for success that we use in our quantitative experi-
learned through logistic regression, from the hand-marked mentsisbasedontheproximityoftherecoveredtorsojoints
half-limbs(Figure5). (hipsandshoulders)togroundtruthdata.
4
7 1
0.9
6
0.8
ns5 0.7
o
m. of detecti34 percentage000...456
nu combined
2 contour 0.3 2 half−limbs
shading 3 half−limbs
shape 0.2 4 half−limbs
1 focus 5 half−limbs
0.1
00 2 4 6 8 10 12 14 16 18 20 00 2 4 6 8 10 12 14 16 18 20
num. of candidates num. of candidates
Figure6:Evaluatinghalf-limbcues:wefixthenum- Figure 7: Evaluating half-limb detection:
beroftopcandidatesgeneratedforeachimageand thepercentageofimageswhichhaveatleast
showtheaveragenumberofhalf-limbdetectionsfor k half-limbs being detected. 89% of the
Figure5: Hand-segmentedlimbsused individual cues and the linear classifier combining imageshaveatleast3half-limbsbeingde-
fortraining. allthecues.Amongthetop8candidates,inaverage tectedamongthetop8candidates.
thereare4.08truepositivedetections.
4.2.1 FindingTorsoswithSegmentations head/torso combination, which specifies an orientation of
thetorsoandthejointpositionsofshouldersandhips.
We use the same Normalized Cuts segmentations to gen- We conductedexperiments using 62 images of baseball
eratecandidatetorsos. Unlikehalf-limbs, atorsotypically players from our set of sports news photographs. Ground
consistsofmorethanonesegment. Givenasegmentation, truth positions of the torso joints were marked by hand, a
we look for all combinations of segments which meet the recoveredtorsoisdeemedtobecorrectifall4torsojoints
scale constraint, i.e., contained in a bounding box of suit- are within 60 pixels of true positions. The results of run-
ablescale. Wethenclassifythesecandidateswithasimilar ning the segmentation-based torso detector are shown on
setofcues: contour,shape,andfocus. Thecontourcueand Figure 8 (solid green line). For reference, the mean head
the focus cue are exactly the same as in the detection of diameterintheimagesisabout50pixels.
half-limbs. Weagainuseasimplerectanglemodelfortorso ComparingwithExemplar-basedApproach: Asfur-
shape.Theshadingcueisdroppedinthiscase,aswedonot therevaluationofthesegmentation-basedtorsodetector,we
expectthetorsostohaveacharacteristicshadingpattern. compare it with an earlier exemplar-based approach. The
PuttingHeadandTorsoTogether: Thesegmentation- exemplar-based torso detector is based on the work in [8].
basedtorsodetectorworkswellinfindingthetrunkofseg- We hand-label torso joint positions in a set of training im-
mentscomprisingthetorso. However,wealsoneedtheori- agestobeusedasexemplars. Exemplarsarerepresentedas
entation of a torso to determine the shoulder and hip posi- collectionsofedges,obtainedusingthetexture-suppressing
tions. There are a variety of poses in the baseball player edgedetectorin[6]. Weusethetechniqueofrepresentative
images,includingsomeplayerslyingparalleltotheground shape contexts to match test images to stored exemplars.
plane. Theorientationisdifficulttoestimatebytorsoitself. Theshapecontextsusedinexperimentshavearatherlarge
Forthisweneedtoputheadandtorsotogether. spatialextent,witharadiusofapproximately 1 oftheheight
3
For each candidate torso and each orientation, we find ofaperson.Informationfromtheconfigurationoftheentire
the best matching head. A candidate head consists of one bodyisbeingusedinthesedescriptors.
orapairofsegments. Thesamesetofcues,contour,shape For the exemplar-based torso detector, a leave-one-out
andfocus,areusedtoevaluatethescoreofacandidatehead. testing strategy was employed. For each test image, we
Theshapemodeloftheheadissimplyadisk,whosescaleis used the other 61 images as the exemplars for matching.
determinedby the candidate torso. Oneobvious extension Resultsforthisexemplar-basedtorsodetectorareshownon
istoaddanoff-the-shelffrontalfacedetectorasfurtherev- Figure8(blackline).Itillustratesthatwiththissetofexem-
idenceforthepresenceofahead. Wehavenotdoneithere, plarsweareunabletocopewiththevariationinappearance
however,sinceitwon’tbetoousefulonourdataset–most amongthedifferentplayers.
players are either not looking at the camera or their faces
areobscuredbytheircaps. 5. Assembling Body Parts
Thetorsoscore,theheadscore,andanothersimplescore
fortheirrelativepositions(aGaussianmodellearnedfrom Using the candidate limbs and torsos from the procedures
hand-labeled data) are then multiplied into a score for the described above, we would like to generate a shortlist of
5
Torso Detectors elbow-to-wrist,so23),thereare:
1
(cid:18) (cid:19)
L
0.9 ·8·7·6·23·T
3
0.8 segments
segments with limbs
ect0.7 exemplars possiblepartialconfigurations. Inourexperiments,Lisbe-
orr tween 5 and 7, and T is about 50, leading to 2-3 million
C
n 0.6 partialconfigurations.
o
Fracti0.5 Of the millions of possible partial configurations enu-
merated above, many are physically impossible due to
0.4 theirviolationofglobalanthropometricandkinematiccon-
0.3 straints. We can efficiently prune the set of partial config-
urations using these constraints while assuming scaled or-
0.2
0 5 10 15 20 25 30 35 40 45 50 thographicprojection. Thespecificconstraintsweuseare:
Shortlist Length
Figure8:Evaluatingthetorsofinders:theX-axisisthenumberof Relative widths: Projection leads to foreshortening of
topcandidatetorsos(lengthof“shortlist”),theY-axisthepercent- thelengthsofbodysegments. However,weareabletoes-
ageofimagesofwhichwefinda“correct”torsointhesecandi- timatethewidthsoftheproposedcandidatesegments. We
dates. Atorsois“correct”ifallthefourjointpositionsofshoul- usetheseestimatedwidths,alongwithanthropometricdata
ders and hips are within a threshold (1 to 1.5 head size) of the toprunecombinationsthathaverelativewidthsmorethan4
ground truth. The segmentation-based detector outperforms the standarddeviationsawayfromtheirestimatedmeans.
exemplar-based detector, mainly because of the limited amount
Length given torso: We also assume that the torso is
of training data. The separately detected limbs provide signifi-
not horribly foreshortened. In our experiments we assume
cantconstraintsfortorsos:thecombinedsegmentation-basedlimb
thatthetorsoformsananglewiththeimageplanebetween
andtorso detectorperforms thebest. For80% performance, the
−40o and 40o. We estimate the length of the torso in
exemplar-basedmethodrequires18torsos,thesegmentationbased
one 6, the combined limb and segmentation one 5. At 90% the the image, then use this assumption and our data on rela-
lengthsare29,17,and10. tivelengthsofsegmentsinturntoyieldconservativeupper
bounds on the lengths of the limbs. Again we prune con-
possiblecompletebodyconfigurations.Asimplemethodof figurations containing a limb 4 standard deviations greater
assemblingconfigurationsistotakeeachtorsoandindepen- thanitsestimatedmeanlength.
dentlyselectthebestlimbtoconnecttothetorsojoints(hips Adjacency: Ifapartialconfigurationcontainsanupper
and shoulders) based on the limb detector outputs. This is limb, it must be adjacent to its corresponding joint on the
akin to the dynamic programming-based approaches used torso. Similarly, if both the upper and lower portions of a
inpreviouswork. limbareset,theymustbeadjacentattheelbow/knee. Fur-
thermore, given upper bounds on limb lengths as above,
However, there are dependencies between the limbs of
constraints exist for solitary lower limbs as well. These
the body that would not be captured in such an approach.
simplekinematicconstraintsareusedtoprunedisconnected
Instead,wereasonaboutwhichlimbcandidatescanandcan
partialconfigurations.
not be used together, using constraints on relative widths
and lengths of limbs, and symmetry in clothing between Symmetry in clothing: The final constraint we use is
certainlimbs. thatthecolorhistograms(measuredinLabspace)ofcorre-
spondingsegments(e.g. leftandrightlowerarms)mustnot
be wildly dissimilar (χ2 distance < 0.3). This often pre-
5.1 EnforcingGlobalConstraints
ventsanarmtobematchedtoapieceofgrass,forexample.
Fixing3half-limbstospecificbodypartsactivatessome
¿From L candidate half-limbs and T candidate torsos, we
subset of these constraints. For our experiments, we se-
generate a set of partial body configurations. A partial
lected a threshold for each of these pruning mechanisms
configuration consists of 3 of the candidate half-limbs (L
suchthatwereducetheoriginal2-3millionconfigurations
choose 3)1, each fixed as one of the 8 body segments in
downtoapproximately1000.
our model (8·7·6), along with one of the T torso candi-
dates. Oncewehavefixedcandidatehalf-limbstospecific
5.2 SortingandCompletingConfigurations
bodyparts, wecaninstantiatethelimbdetectorsdescribed
above. Since each candidate half-limb also needs an as- Afterthepruningphasewehaveasetofpartialbodyconfig-
sociated “polarity” (e.g. for lower arm, wrist-to-elbow or urations,eachconsistingofatorsoalongwith2or3limbs
(assembledfrom3half-limbs). Witheachoftheseconfigu-
1Thenumber3ischosenbasedonFigure7. rations,therearetheoutputsoftheassociatedlimbdetectors
6
the context that an entire exemplar gives. We believe that
combiningthesetwoapproachesinasensiblemannerisan
importanttopicforfuturework.
Acknowledgments:WethankJaetyEdwardsforhelpin
collecting the dataset. This work was supported by ONR
grant N00014-01-1-0890 (MURI), NSF ITR grant IIS-00-
85864,andaDigitalLibraryGrant1R1-9411334.
References
[1] P.F.FelzenszwalbandD.P.Huttenlocher. Efficientmatch-
ingofpictorialstructures.InProc.IEEEComput.Soc.Conf.
Comput.VisionandPatternRecogn.,2000.
[2] D.M.Gavrila. Thevisualanalysisofhumanmovement: A
survey. ComputerVisionandImageUnderstanding:CVIU,
73(1):82–98,1999.
[3] D.Hogg. Model-basedvision: Aprogramtoseeawalking
person. ImageandVisionComputing,1(1):5–20,1983.
Figure9:Examplesoftop-rankedtorsos. [4] S. Ioffe and D. Forsyth. Probabilistic methods for finding
people.Int.JournalofComputerVision,43(1):45–68,2001.
andtorsodetectors. Weassignthefinalscoreforaconfig- [5] J.Malik,S.Belongie,T.Leung,andJ.Shi.Contourandtex-
uration to be a linear combination of these limb and torso tureanalysisforimagesegmentation. Int.JournalofCom-
puterVision,43:7–27,2001.
scores. The results of using this combined score as a new
[6] D.Martin,C.Fowlkes,andJ.Malik.Learningtofindbright-
torsodetectorarealsoshowninFigure8andFigure9.
nessandtextureboundariesinnaturalimages. NIPS,2002.
To go from partial configurations (with missing limbs) [7] A.Mohan,C.Papageorgiou,andT.Poggio. Example-based
to the full human figures, we search for entire limbs off object detection in images by components. IEEE Trans.
PAMI,23(4):349–361,2001.
ofemptytorsojoints. Theresultingfullconfigurationsare
[8] G. Mori and J. Malik. Estimating human body configura-
shown in Figure 1 and Figure 10. The configurations are
tions using shape context matching. In European Confer-
orderedbytheircombinedlimbandtorsoscore. Figure10 enceonComputerVisionLNCS2352,volume3,pages666–
showsoneselectedentry(anditscorrespondingrank)onthe 680,2002.
[9] NIST. Anthrokids - anthropometric data of children,
final shortlist of body configurations for each image. The
http://ovrt.nist.gov/projects/anthrokids/,1977.
problemofautomaticallyevaluatingtheseshortlistsofcon-
[10] X. Ren and J. Malik. Learning a classification model for
figurationstofindthecorrectoneisachallengingproblem, segmentation. InProc.9thInt.Conf.ComputerVision,vol-
andisnotaddressedhere. ume1,pages10–17,2003.
[11] S. Russell and P. Norvig. Artificial Intelligence: A Mod-
6. Discussion ernApproach. Prentice-Hall,EnglewoodCliffs,NJ,second
edition,2003.
In this work we have demonstrated how to use low-level [12] G.Shakhnarovich,P.Viola,andT.Darrell. Fastposeesti-
mationwithparametersensitivehashing. InProc.9thInt.
segmentationtodriverecognition. Segmentsandsuperpix-
Conf.ComputerVision,volume2,pages750–757,2003.
elsgeneratedbytheNormalizedCutsalgorithmareusedto
[13] Y.Song,L.Goncalves,andP.Perona. Unsupervisedlearn-
proposecandidatesforlimbsandtorsos. Thesecandidates ingofhumanmotion. IEEETrans.PAMI,25(7):814–827,
arethenverifiedusingavarietyofcues. Afterthat,finding 2003.
[14] J.SullivanandS.Carlsson. Recognizingandtrackinghu-
consistentbodyconfigurationsbecomesaConstraintSatis-
man action. In European Conference on Computer Vision
faction Problem [11]. While not pursued in this paper, it
LNCS2352,volume1,pages629–644,2002.
would be straightforward to apply techniques from the AI [15] A.Torralba,K.P.Murphy,W.T.Freeman,andM.A.Rubin.
heuristicsearchliterature(e.g.,best-firstsearchandA∗)to Context-based vision system for place and object recogni-
tion. InICCV,pages273–280,2003.
reducethecomputationalcomplexity.
[16] K. Toyama and A. Blake. Probabilistic exemplar-based
The problem of recovering human body configurations
trackinginametricspace. InProc.8thInt.Conf.Computer
inageneralsettingisarguablythemostdifficultrecognition Vision,volume2,pages50–57,2001.
problem in computer vision. By no means do we claim to [17] Z.Tu,X.Chen,A.Yuille,andS.Zhu. Imageparsing: seg-
mentation,detection,andrecognition.InProc.9thInt.Conf.
havesolvedithere;muchworkstillremainstobedone. An
ComputerVision,pages18–25,2003.
approachsuchasours,basedonassemblingconfigurations
[18] P.Viola,M.Jones,andD.Snow.Detectingpedestriansusing
fromdetectedsalientparts,hastheadvantageof“composi- patternsofmotionandappearance. InProc.9thInt.Conf.
tionality”–thereisnoneedtostoretemplatesforallpossi- ComputerVision,pages734–741,2003.
[19] S.X.YuandJ.Shi. Object-specificfigure-groundsegrega-
bleconfigurationsasinanexemplar-basedapproach. How-
tion. InProc.IEEEComput.Soc.Conf.Comput.Visionand
ever,thismodularitycomesatadefinitecost.Detectingand
PatternRecogn.,volume2,pages39–45,2003.
localizing parts is more reliably done when provided with
7
(a) (b) (c) (d)
Figure10: Selectedresultsfromshortlistoffinalconfigurations. (a)Inputimage, (b)Candidatehalf-limbs, (c)Extractedbodyconfig-
urations, (d)Associatedsegmentation. Onebodyconfigurationfromtheshortlistforeachimageisshown. Shortlistrankingsforeach
configuration,rowstoptobottom:(1st,1st,11th,1st,18th,1st,3rd,3rd,1st)
8