Real-Time Reconstruction of Static and Dynamic Scenes

196 Pages·2015·31.25 MB·English

by  Michael Zollhöfer

Real-Time Reconstruction
of Static and Dynamic Scenes
Der Technischen Fakultät
der Friedrich-Alexander-Universität
Erlangen-Nürnberg
zur
Erlangung des Grades Dr.-Ing.
vorgelegtvon
Michael Zollhöfer
ausHerzogenaurach
AlsDissertationgenehmigt
vonderTechnischenFakultät
derFriedrich-Alexander-UniversitätErlangen-Nürnberg
TagdermündlichenPrüfung: 22.12.2014
VorsitzendedesPromotionsorgans: Prof.Dr.-Ing.habil.MarionMerklein
Gutachter: Prof.Dr.GüntherGreiner
Prof.Dr.Christianeobalt
Revision1.00
©2014,CopyrightMichaelZollhöfer
[email protected]
AllRightsReserved
AlleRechtevorbehalten

Abstract
WiththereleaseoftheMicrosoXbox360Kinect,anaffordablereal-time
RGB-Dsensorisnowavailableonthemassmarket. ismakesnewtech-
niques and algorithms, which have previously been only available to re-
searchersandenthusiasts, accessibleforaneverydayusebyabroadaudi-
ence. Applicationsrangefromtheacquisitionofdetailedhigh-qualityre-
constructionsofeverydayobjectstotrackingthecomplexmotionsofpeo-
ple.Inaddition,thecaptureddatacanbedirectlyexploitedtobuildvirtual
reality applications, i.e. virtual mirrors, and can be used for gesture con-
trolofdevicesandmotionanalysis.Tomaketheseapplicationseasy-to-use
inoureverydaylife, theyshouldbeintuitivetocontrolandprovidefeed-
backatreal-timerates.Inthisdissertation,wepresentnewtechniquesand
algorithmsforbuildingthree-dimensionalrepresentationsofarbitraryob-
jectsusingonlyasinglecommodityRGB-Dsensor, manuallyeditingthe
acquiredreconstructionsandtrackingthenon-rigidmotionofphysically
deforming objects at real-time rates. We start by proposing the use of a
statisticalpriortoobtainhigh-qualityreconstructionsofthehumanhead
usingonlyasinglelow-qualitydepthframeofacommoditysensor.Weex-
tendthisapproachandobtainevenhigherqualityreconstructionsatreal-
timeratesbyexploitingallinformationofacontiguousRGB-Dstreamand
jointly optimizing for shape, albedo and illumination parameters. ere-
aer,weshowthatamovingsensorcanbeusedtoobtainsuper-resolution
reconstructionsofarbitraryobjectsatsensorratebyfusingalldepthobser-
vations. Wepresentstrategiesthatallowustohandleavirtuallyunlimited
reconstructionvolumebyexploitinganewsparsescenerepresentationin
combinationwithanefficientstreamingapproach. Inaddition,wepresent
ahandlebaseddeformationparadigmthatallowstheusertoeditthecap-
turedgeometry,whichmightconsistofmillionsofpolygons,usinganinter-
activeandintuitivemodelingmetaphor. Finally,wedemonstratethatthe
motionofarbitrarynon-rigidlydeformingphysicalobjectscanbetracked
atreal-timeratesusingacustomhigh-qualityRGB-Dsensor.
i

Acknowledgements
is dissertation captures the compressed technical contributions and al-
gorithmicachievementsoffoursolidyearsofwork. Istartedmyworkin
January2011withastronginterestingeometryprocessing, optimization
problems, rendering and GPGPU programming, but quite clueless about
a suitable field of research. Little did I know that all my rather different
interestswouldmagicallylineup... afewyearslater.
isallwouldnothavebeenpossiblewithouttheconstantsupport,friendly
encouragements and selfless contributions of my colleagues and cowork-
ers.MysupervisorGüntherGreineralwayssupportedmeandgavemethe
freedom to fully pursue my own research interests. Jochen Süßmuth in-
troducedmetotheworldofnon-rigidregistrationandsupervisedmedur-
ingmystudiesandinthefirstyearofmyPhD.MatthiasNießnerhelped
me to rediscover my will to conduct research and fueled my curiosity in
online 3D reconstruction methods. Marc Stamminger helped during all
deadlineswithhisadviceandcraedalotofamazingillustrations. Frank
Bauer, our Blender guru, shared his tricks and helped with the produc-
tion of videos. I am thankful for all this support and feel in great debt.
Whatimpressedmemostisthatallofmycolleaguesmanagedtoendure
myconstantrantsaboutthealgorithmsthatbreakunderrealworldcondi-
tions:ChristianSiegl,KaiSelgrad,MagdalenaPrus,MichaelMartinek,Jan
Kretschmer, Quirin Meyer, Roberto Grosso, Matteo Colaianni, Matthias
Innmann, Henry Schäfer, Benjamin Keinert, Franziska Bertelshofer and
ChristophWeber. I feelprivileged for theopportunity towork in such a
greatandcreativeenvironmentwithpeoplethathavebecomemorethan
justcolleaguestomeoverthelastfouryears.
Iwouldalsoliketothankallofmystudentsthatworkedwithmeduring
theirBachelorandMasterprojects. Togetherwelearnedalotandgotex-
posedtonewinterestingtopicsandideas. EzgiSerthelpeddevelopingthe
presentedlatticebaseddeformationapproach. Justusieshelpedmesub-
stantially with my first DFG project and the presented interactive model
iii
basedreconstructionmethod.Iamhappythathewillcontinuemyresearch
atthechair.
ankstoShahramIzadiforgivingmetheopportunitytoconductresearch
in his group at Microso Research Cambridge. I met a lot of cool peo-
pleduringmystay:AndrewFitzgibbon,ChristophRhemann,Christopher
Zach,DavidKim,CemKeskinandSeanFanello.Itwasathrillingtimefull
ofhardwork,excellentfoodandgreatparties.
ankstoMatthiasNießner,MatthewFisher,ChengleiWuandChristian
eobalt for helping with my real-time non-rigid reconstruction project
andAngelaDaiforthenicevoiceover.
IamalsogratefultotheGermanResearchFoundation(DFG)forfunding
myworkoverthelastfouryearsundergrantSTA-662/3--1andGRK-1773.
Lastbutnotleast,thankstomyparentsKerstinandFranzforsupporting
meonallofmyendeavors<3.
October 2014
MZ
iv
Contents
1 MotivationandFundamentals 1
1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Real-TimeRGB-DSensors. . . . . . . . . . . . . . . . . . 2
1.3 GeneralPurposeGPUProgramming . . . . . . . . . . . . 7
2 OptimizationTheory 11
2.1 ModelFitting . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.2 LeastSquaresOptimization . . . . . . . . . . . . . . . . . 13
2.3 LinearLeastSquaresOptimization . . . . . . . . . . . . . 15
2.4 Non-linearLeastSquaresOptimization . . . . . . . . . . . 16
3 ContributionandOutline 19
I ReconstructionofPersonalizedAvatars 23
4 Introduction 25
5 RelatedWork 27
6 Method 29
6.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
6.2 DataAcquisition . . . . . . . . . . . . . . . . . . . . . . . 29
6.2.1 DataPreparation . . . . . . . . . . . . . . . . . . 30
6.2.2 FaceandFeatureDetection . . . . . . . . . . . . . 31
6.2.3 FaceSegmentation. . . . . . . . . . . . . . . . . . 33
6.3 FittingaGenericFaceModel . . . . . . . . . . . . . . . . 33
6.3.1 FittingEnergyTerm . . . . . . . . . . . . . . . . . 35
6.3.2 RegularizationEnergyTerm . . . . . . . . . . . . 36
6.3.3 Optimization . . . . . . . . . . . . . . . . . . . . 36
6.3.4 Fittingthemorphablefacemodel . . . . . . . . . . 37
v
Contents
7 Results 39
8 Conclusion 41
II ModelbasedReconstructionoftheHumanHead 43
9 Introduction 45
10 RelatedWork 47
10.1 Model-free3D-Reconstruction . . . . . . . . . . . . . . . 47
10.2 Model-based3D-Reconstruction . . . . . . . . . . . . . . 48
11 Method 51
11.1 PipelineOverview . . . . . . . . . . . . . . . . . . . . . . 51
11.2 HeadPoseEstimation . . . . . . . . . . . . . . . . . . . . 52
11.3 DataFusion . . . . . . . . . . . . . . . . . . . . . . . . . . 53
11.4 EstimatingModelParameters . . . . . . . . . . . . . . . . 54
11.4.1 StatisticalShapeModel . . . . . . . . . . . . . . . 54
11.4.2 ObjectiveFunction . . . . . . . . . . . . . . . . . 55
11.4.3 ParameterInitialization . . . . . . . . . . . . . . . 56
11.4.4 JointNon-LinearGPUOptimizer . . . . . . . . . 57
12 Results 59
12.1 RuntimeEvaluation . . . . . . . . . . . . . . . . . . . . . 59
12.2 ReconstructionQuality . . . . . . . . . . . . . . . . . . . 60
12.3 Applications . . . . . . . . . . . . . . . . . . . . . . . . . 60
13 Conclusion 65
III Online3DReconstructionatScale 67
14 Introduction 69
15 Relatedwork 71
vi