Techniques for Binocular Markerless Visual Tracking of 3D Articulated Bodies

212 Pages·2016·6.87 MB·English

Techniques for Binocular Markerless Visual Tracking of 3D Articulated Bodies
AndrewW.B.Smith
B.Eng. (Elec)
B.Bus. Man. (Economics)
AthesissubmittedforthedegreeofDoctorofPhilosophyat
TheUniversityofQueenslandin2016
SchoolofInformationTechnologyandElectricalEngineering
ii
Abstract
This thesis advances methods for performing markerless visual tracking of articulated bodies using
oneortwocameras. TheresearchpresentedaimstoimproveuponexistingBayesianinspiredtracking
methods, by examining the ‘building blocks’ of these tracking algorithms, in particular the measure-
ment function design, the state space selection, and local optimization methods. Results presented
in this thesis show that improvements can be made in all of these areas. These improvements are
applicabletoavarietyofBayesiantrackingalgorithms.
This thesis begins by examining literature relevant to the visual tracking problem. This includes the
measurement functions used by other authors, focussing on the edge detection methods used in both
tracking and segmentation problems. A general overview of the global search problem is given next,
asaglobalsearchisafundamentalpartofaBayesiantrackingalgorithm. ThecombinationofNewton
like local optimization methods and the measurement functions used in visual tracking problems is
examined next, and it is shown that Newton optimizers are not ideally suited to these measurement
functions. The Bayesian tracking framework is then detailed, along with a review of several existing
Bayesiantrackingalgorithms. FinallysomenonBayesiantrackingalgorithmsarediscussed.
Following the literature review, details of the models used in the experiments presented in this the-
sis are given. These include the articulated human body model, the camera model, image gradient
metrics,selfocclusiontreatment,andagenericcolourbasedregionmeasurementmethod.
Theuseofgraphbasedapproachesforedgemeasurementsistheninvestigated. Graphbasedmethods
are commonly used in image segmentation problems, however have not been applied to visual track-
ing problems. A novel method for performing edge measurements using the ‘shortest path’ around
theobject’soccludingcontourispresented. Unlikeinthesegmentationproblem,selfocclusionmod-
els mean the weights or costs of some graph vertices can not be determined. Different treatments
for occluded graph vertices are given and evaluated. It is shown that the graph based approach pro-
duces observational likelihoods that are more accurate and have significantly fewer local maxima
than the edge measurement schemes previously used in tracking problems. While this approach is
computationally more expensive than other methods, it is argued that this is offset by the reduced
computationalexpenseoftheglobalsearchprocedureusedintrackingalgorithms.
The choice of state space used in the tracking problems is examined next. While most authors have
used a state space based on the joint angles of the human body, a Cartesian state space based on
the world coordinates of limbs is proposed. While Cartesian based state spaces have been used by
iii
other authors for representations of kinematic models, to the author’s knowledge they have not been
used for full kinematic models. It is shown that that the more linear relationship between state vari-
ables in the Cartesian space and the 3D locations of sampled points on the object improves dynamic
model predictions and principal component analysis. It is also shown that the Cartesian formulation
also increases the linearity between state variables and the image coordinates of sampled points on
the object. This in turn improves the performance of local optimization methods which make local-
ized quadratic approximations to the measurement function. While the Cartesian based space has a
higher dimensionality than the rotation based space, the geometrically plausible region of the Carte-
sian space has the same content (area) as the rotation space, which negates the well known ‘curse
of dimensionality’. A simple method is given to project an implausible Cartesian state to a geomet-
rically plausible state, as well as a method to dampen the measurement function curvature in these
implausibledirections.
Following this, a novel local optimization method is proposed. This optimization method is specific
to visual tracking problems, and uses the camera geometry to infer interesting search directions.
Treatments for choosing these search directions are given for both the monocular and two camera
cases. A problem decomposition is also used to reduce the computational cost of the optimizer. This
methodisshowntooutperformNewtonbasedoptimizationsinarotationbasedstatespace,andgives
atworstequivalentresultstoaNewtonbasedapproachinaCartesianstatespace,butatasignificantly
reducedcomputationalcost.
Finally, tracking results are presented for a difficult image sequence using the combined ideas pre-
sented in this thesis. This sequence is a golfer performing a golf swing, which is a highly dynamic
motionwithlargeobjectvelocitiesandaccelerations.
iv
Declarationbyauthor
Thisthesisiscomposedofmyoriginalwork,andcontainsnomaterialpreviouslypublishedorwritten
by another person except where due reference has been made in the text. I have clearly stated the
contributionbyotherstojointly-authoredworksthatIhaveincludedinmythesis.
Ihaveclearlystatedthecontributionofotherstomythesisasawhole,includingstatisticalassistance,
survey design, data analysis, significant technical procedures, professional editorial advice, and any
other original research work used or reported in my thesis. The content of my thesis is the result of
work I have carried out since the commencement of my research higher degree candidature and does
not include a substantial part of work that has been submitted to qualify for the award of any other
degree or diploma in any university or other tertiary institution. I have clearly stated which parts of
mythesis,ifany,havebeensubmittedtoqualifyforanotheraward.
I acknowledge that an electronic copy of my thesis must be lodged with the University Library and,
subject to the policy and procedures of The University of Queensland, the thesis be made available
for research and study in accordance with the Copyright Act 1968 unless a period of embargo has
beenapprovedbytheDeanoftheGraduateSchool.
Iacknowledgethatcopyrightofallmaterialcontainedinmythesisresideswiththecopyrightholder(s)
of that material. Where appropriate I have obtained copyright permission from the copyright holder
toreproducematerialinthisthesis.
v
Publicationsduringcandidature
Peerreviewedpublications:
A.W.B.SmithandB.C.Lovell,“VisualTrackingforSportsApplications,”inAPRSWorkshop
•
on Digital Image Computing, B. C. In Lovell and A. J. Maeder, Eds., vol. 1(1), Brisbane,
February2005,pp. 79–84.
IncorporatedintoChapter1definingthescopeandtheaimsofthisthesis.
A.W.B.SmithandB.C.Lovell,“MeasurementFunctionDesignforVisualTrackingApplica-
•
tions,”inProceedingoftheEighteenthInternationalConferenceonPatternRecognition,vol.1,
2006,pp. 789–792.
Incorporated into Section 4.3 examining the construction of the ‘cost trellis’, as well as the
quantitative evaluation of the performance of various edge measurement schemes presented in
Section4.8.5.
A. W. B. Smith, “Self Occlusions and Graph Based Edge Measurement Schemes for Visual
•
TrackingApplications,”AwardedbeststudentpaperatDigitalImagingComputing: Techniques
andApplications,Dec2009.
IncorporatedintoSection4.4examiningtheconstructionofthe‘occludingcontourcosttrellis’,
aswellasthequantitativeevaluationoftheperformanceofvariousedgemeasurementschemes
presentedinSection4.8.
A.W.B.SmithandB.C.Lovell,“AutonomousSportsTrainingfromVisualCues,”inProceed-
•
ingsoftheEighthAustralianandNewZealandConferenceonIntelligentInformationSystems,
B. C. In Lovell, D. A. Campbell, C. B. Fookes, and A. J. Maeder, Eds., vol. 1(1), Sydney,
December2003,pp. 279–284.
X. Zhang, H. Wang, A. W. B. Smith, and B. C. Lovell, “Corner Detection Based on Gradient
•
Correlation Matrices of Planar Curves,” in Pattern Recognition, vol. 43(4), 2010, pp 1207-
1223.
Publicationsincludedinthisthesis
Nopublicationsincluded.
vi
Contributionsbyotherstothethesis
Nocontributionsbyothers.
Statementofpartsofthethesissubmittedtoqualifyfortheawardofanotherdegree
None.
vii
Acknowledgements
There are many people who provided me with much support over the course of my dissertation.
Firstly, I would like to thank Rick Baker for his vision of creating autonomous sports training tech-
nologies,towardswhichthisdissertationwasundertaken. Iwouldalsoliketothankhimforproviding
the equipment necessary for the completion of this dissertation through his company Hi-Tech Video.
FurthermoreIwouldliketothankhimforkeepingmymotivationlevelshighbyremindingmeofthe
manypotentialusesmyresearchcouldhave.
I would like to thank my supervisor Professor Brian Lovell, for his teaching of the undergraduate
course in signals and image processing at the University of Queensland (UQ), which sparked my
interest in the field of computer vision. I would also like to thank him for introducing me to Rick
Baker, which led to me undertaking this research. Lastly I would like to thank him for accepting
me as his postgraduate student. I would also like my colleagues and co-workers at the School of
Information Technology and Electrical Engineering at the University of Queensland. Thanks to the
IRIS research group, Simon, Stefan, Carlos, Ben, Emanual, Christian and Andrew Mehnert, for your
friendship and technical suggestions. Thanks also to Wayne Wilson for providing a productive and
friendlyworkingenvironmentduringmyemploymentinmyvariousrolesworkingforyou.
A special thanks also goes to Andrew Bradley for motivating me to tackle my thesis corrections,
withoutwhichthisthesiswouldnothavebeencompleted.
Thisresearchcouldnothavebeenperformedwithoutfinancialsupportfromseveralsources. Iwould
liketothankRickBakerandHi-TechVideo,andtheSchoolofInformationTechnologyandElectrical
Engineering at the University of Queensland, for funding my scholarship without which I would not
have completed this dissertation. I would also like to thank the Queensland state government for
payingformyuniversitytuitionfeesduringthecourseofmycandidature.
viii
Keywords
Humanmotionestimation,humanbodytracking,visualtracking,measurementfunctiondesign,state
spaceselection,optimization,camerageometry,generativemodels,particlefiltering
AustralianandNewZealandStandardResearchClassifications(ANZSRC)
ANZSRCcode: 080104,ComputerVision,100%.
FieldsofResearch(FoR)Classification
FoRcode: 0801,ArtficialIntelligenceandimageProcessing,100%.
Contents
ListofFigures xv
ListofTables xix
1 Introduction 1
1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Aim . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.3 Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.4 EvaluationofResults . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.5 OrganizationofthisThesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.6 OriginalContributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.6.1 MeasurementFunctionDesign . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.6.2 StateSpaceSelection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.6.3 LocalOptimizationMethod . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2 LiteratureReview 11
2.1 MeasurementMethods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.1.1 ImageDerivativesandEdgeDetection . . . . . . . . . . . . . . . . . . . . 12
2.1.2 EdgeBasedMeasurementFunctions . . . . . . . . . . . . . . . . . . . . . 13
2.1.3 GraphTheoreticMethodsandEdge/BoundaryDetection . . . . . . . . . . . 17
2.1.4 RegionBasedMeasurementMethods . . . . . . . . . . . . . . . . . . . . . 19
2.2 SearchandOptimizationOverview . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.3 Newton-likeLocalOptimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.3.1 HessianDampening . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.3.2 TrustRegions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
ix
x CONTENTS
2.3.3 Quasi-NewtonApproaches . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.3.4 SuitabilityofNewton-LikeLocalOptimizationforvisualtracking . . . . . . 24
2.4 Bayesian(Generative)TrackingAlgorithms . . . . . . . . . . . . . . . . . . . . . . 27
2.4.1 TheKalmanFilter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.4.2 TheExtendedandUnscentedKalmanFilters . . . . . . . . . . . . . . . . . 29
2.4.3 ParticleFiltering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
2.4.4 AnnealedParticleFiltering . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
2.4.5 HyperdynamicAnnealedParticleFiltering . . . . . . . . . . . . . . . . . . 34
2.4.6 TransitionPointRoadMaps . . . . . . . . . . . . . . . . . . . . . . . . . . 36
2.4.7 CovarianceScaledSampling(CSS) . . . . . . . . . . . . . . . . . . . . . . 37
2.4.8 KinematicJumpSampling . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
2.4.9 ParallelFilterBanks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
2.5 DiscriminativeTrackingAlgorithms . . . . . . . . . . . . . . . . . . . . . . . . . . 40
2.5.1 MappingsfromSilhouetteSpaces . . . . . . . . . . . . . . . . . . . . . . . 40
2.5.2 VolumetricTracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
2.5.3 StrongMotionModels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
2.5.4 HistogramofOrientedGradients . . . . . . . . . . . . . . . . . . . . . . . 42
3 Modelling 43
3.1 HumanBodyModelling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
3.1.1 DegreesofFreedom . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
3.1.2 LinkSurfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
3.1.3 SelfOcclusionModel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
3.2 CameraModel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
3.3 ImageGradientMetricandEdgeDetection . . . . . . . . . . . . . . . . . . . . . . 53
3.4 Learningfeaturemodelsalongmeasurementlines . . . . . . . . . . . . . . . . . . . 55
3.5 RegionBasedMeasurement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
3.5.1 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
3.5.2 RegionConsistencyMeasurement . . . . . . . . . . . . . . . . . . . . . . . 57
4 MeasurementFunctionDesign 59
4.1 WhatConstitutesaGoodMeasurementFunction? . . . . . . . . . . . . . . . . . . 60
4.2 GraphBasedMethodsforEdgeMeasurement . . . . . . . . . . . . . . . . . . . . . 61
4.2.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62