Deformable and Articulated 3D Reconstruction from monocular video sequences

212 Pages·2012·9.11 MB·English

1
Deformable and Articulated 3D Reconstruction
from monocular video sequences
Marco Paladini
QueenMary, UniversityofLondon
Submitted for the degree of Doctor of Philosophy
Queen Mary, University of London
2012
2
3
Deformable and Articulated 3D Reconstruction
from monocular video sequences
Marco Paladini
QueenMary, UniversityofLondon
Abstract
This thesis addresses the problem of deformable and articulated structure from motion from
monocular uncalibrated video sequences. Structure from motion is defined as the problem of
recoveringinformationaboutthe3Dstructureofscenesimagedbyacamerainavideosequence.
Ourstudyaimsatthechallenging problemofnon-rigid shapes(e.g. abeatingheartorasmiling
face). Non-rigid structures appear constantly in our everyday life, think of a bicep curling, a
torso twisting or a smiling face. Our research seeks a general method to perform 3D shape
recovery purely from data, without having to rely on a pre-computed model or training data.
Openproblems inthefieldarethedifficultyofthenon-linear estimation, thelackofareal-time
system, large amounts of missing data in real-world video sequences, measurement noise and
strong deformations. Solving these problems would take us far beyond the current state of the
art in non-rigid structure from motion. This dissertation presents our contributions in the field
of non-rigid structure from motion, detailing a novel algorithm that enforces the exact metric
structure of the problem at each step of the minimisation by projecting the motion matrices
onto the correct deformable or articulated metric motion manifolds respectively. An important
advantage of this new algorithm is its ability to handle missing data which becomes crucial
when dealing with real video sequences. We present a generic bilinear estimation framework,
whichimprovesconvergenceandmakesuseofthemanifoldconstraints. Finally,wedemonstrate
a sequential, frame-by-frame estimation algorithm, which provides a 3D model and camera
parametersforeachvideoframe,whilesimultaneouslybuildingamodelofobjectdeformations.
Submitted for the degree of Doctor of Philosophy
Queen Mary, University of London
2012
4
5
Contents
1 Introduction 15
1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
1.2 Therigidcase . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
1.3 NonRigidStructurefromMotion . . . . . . . . . . . . . . . . . . . . . . . . 18
1.4 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
1.5 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
1.6 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2 LiteratureReview 27
2.1 Approaches to3Dshapereconstruction . . . . . . . . . . . . . . . . . . . . . 28
2.1.1 Shape-from-X . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.2 RigidStructurefromMotion . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
2.2.1 Uncalibrated case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
2.2.2 Bundleadjustment . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
2.2.3 Factorisation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
2.3 Non-RigidStructurefromMotion . . . . . . . . . . . . . . . . . . . . . . . . 44
2.3.1 Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
2.3.2 Bregleretal.’sOriginalNon-RigidFactorisation Algorithm . . . . . . 47
2.3.3 Anill-posed problem . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
2.4 ATaxonomyofNon-RigidShapeEstimationfromMonocularSequences . . . 53
2.5 NRSfMwiththelinearlow-rankbasisshapemodel . . . . . . . . . . . . . . . 56
2.6 Closed-form SolutionstoNRSfM . . . . . . . . . . . . . . . . . . . . . . . . 57
6 Contents
2.6.1 Basisconstraints: Xiao-Chai-Kanade . . . . . . . . . . . . . . . . . . 57
2.6.2 Closedformsolution forperspective cameras: Hartley-Vidal . . . . . . 60
2.6.3 Brand’sdirectmethod . . . . . . . . . . . . . . . . . . . . . . . . . . 60
2.7 NRSfMvianon-linear optimisation . . . . . . . . . . . . . . . . . . . . . . . 61
2.7.1 Alternation methods . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
2.7.2 Non-rigid bundleadjustment . . . . . . . . . . . . . . . . . . . . . . . 65
2.8 Trainedmodelsfornon-rigid shapeanalysis . . . . . . . . . . . . . . . . . . . 67
2.9 Reconstruction withMissingData . . . . . . . . . . . . . . . . . . . . . . . . 69
2.10 Alternativeshapemodels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
2.10.1 Piecewisereconstruction methods . . . . . . . . . . . . . . . . . . . . 71
2.10.2 ManifoldLearning . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
2.10.3 Reconstruction inTrajectorySpace . . . . . . . . . . . . . . . . . . . 75
2.11 Template-based methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
2.12 Articulated StructurefromMotion: A-SfM . . . . . . . . . . . . . . . . . . . 81
2.12.1 Articulated ShapeModel . . . . . . . . . . . . . . . . . . . . . . . . . 81
2.12.2 Subspaceanalysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
2.12.3 Jointestimationin3D . . . . . . . . . . . . . . . . . . . . . . . . . . 84
2.13 Closure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
3 MetricProjections forDeformableandArticulatedStructure-From-Motion 89
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
3.1.1 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
3.2 Factorisation forStructurefromMotion . . . . . . . . . . . . . . . . . . . . . 94
3.2.1 RigidShape . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
3.2.2 DeformableShapeModel . . . . . . . . . . . . . . . . . . . . . . . . 95
3.2.3 Articulated ShapeModel . . . . . . . . . . . . . . . . . . . . . . . . . 96
3.3 MetricUpgrade . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
7
3.3.1 MetricProjection: DeformableCase . . . . . . . . . . . . . . . . . . . 101
3.3.2 Convexrelaxation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
3.3.3 NewtonmethodontheStiefelmanifold . . . . . . . . . . . . . . . . . 104
3.3.4 MetricProjection: Articulated Case . . . . . . . . . . . . . . . . . . . 106
3.4 Reconstruction withMissingData . . . . . . . . . . . . . . . . . . . . . . . . 109
3.5 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
3.5.1 DeformableStructure . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
3.5.2 ArticulatedStructure . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
3.6 Summaryanddiscussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
4 BilinearmodellingviaAugmentedLagrangeMultipliers(BALM) 133
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
4.2 RelatedWork . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
4.3 Problemstatement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
4.4 TheBALMalgorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
4.4.1 Solvingforthemanifoldconstraints . . . . . . . . . . . . . . . . . . . 139
4.4.2 Solvingforthebilinearfactorisation . . . . . . . . . . . . . . . . . . . 141
4.4.3 Solvingforthemissingdata . . . . . . . . . . . . . . . . . . . . . . . 142
4.4.4 Initialisation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
4.5 Example1: BALMforRigidandNon-RigidSfM . . . . . . . . . . . . . . . . 142
4.6 Example2: BALMforArticulated SfM . . . . . . . . . . . . . . . . . . . . . 144
4.6.1 Articulatedmanifoldprojector . . . . . . . . . . . . . . . . . . . . . . 145
4.7 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
4.7.1 Syntheticexperiments: NRSfM . . . . . . . . . . . . . . . . . . . . . 145
4.7.2 Realdata: NRSfM . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
4.7.3 Realdata: RigidSfM . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
4.7.4 Realdata: Articulated SfM . . . . . . . . . . . . . . . . . . . . . . . . 150
8 Contents
4.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
5 Sequentialnon-rigidstructurefrommotion 153
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154
5.2 RelatedWork . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
5.3 NewDeformation Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
5.3.1 ClassicalExplicitLow-RankShapeModel . . . . . . . . . . . . . . . 156
5.3.2 Proposed3D-ImplicitLow-RankShapeModel . . . . . . . . . . . . . 158
5.4 ASequentialApproachtoNRSfM . . . . . . . . . . . . . . . . . . . . . . . . 160
5.5 CameraTrackingGivenaKnownModelV . . . . . . . . . . . . . . . . . . . 161
5.5.1 Initialisation: LinearEstimationofU andR . . . . . . . . . . . . . . 164
f f
5.6 Sequential UpdateoftheShapeModel . . . . . . . . . . . . . . . . . . . . . . 165
5.6.1 RankIncreaseCriterion . . . . . . . . . . . . . . . . . . . . . . . . . 166
5.6.2 ModelUpdate: EstimatingNewRowofVandNewColumnofU . . . . 166
5.6.3 Bootstrapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
5.7 Limitingtherank . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
5.7.1 Modelcompression . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
5.8 Missingdata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
5.9 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
5.9.1 Motioncapturesequence CMU-face . . . . . . . . . . . . . . . . . . . 170
5.9.2 RealData . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
5.9.3 Missingdata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
5.10 Application toModel-based featuretracking . . . . . . . . . . . . . . . . . . . 173
5.10.1 Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
5.10.2 Forwardmodel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
5.10.3 Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178
5.10.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
9
5.11 Summaryandcritique . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
6 Conclusions 185
6.1 Non-rigidStructurefromMotionusingMetricProjections . . . . . . . . . . . 185
6.2 BilinearproblemsinComputerVision . . . . . . . . . . . . . . . . . . . . . . 186
6.3 Thechallenge ofreal-timeestimation . . . . . . . . . . . . . . . . . . . . . . 187
6.4 Futurework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188
Bibliography 189
A Optimization,deformablecase 205
B Convexrelaxation, Articulated Case 207
10