TEXT ANNOTATION AND SIMILARITY SEARCH YIN Yifang A THESIS SUBMITTED FOR THE ...

183 Pages·2015·6.53 MB·English

GEO-REFERENCED VIDEO RETRIEVAL:
TEXT ANNOTATION AND SIMILARITY SEARCH
YIN Yifang
(B.Sc., Northeastern University, 2011)
A THESIS SUBMITTED
FOR THE DEGREE OF DOCTOR OF PHILOSOPHY
SCHOOL OF COMPUTING
NATIONAL UNIVERSITY OF SINGAPORE
2016

DECLARATION
I hereby declare that this thesis is my original work and it has been written
by me in its entirety. I have duly acknowledged all the sources of information
which have been used in the thesis.
This thesis has also not been submitted for any degree in any university
previously.
YIN Yifang 18 Jan. 2016

ACKNOWLEDGEMENTS
This dissertation is a summary of my four years research work. I would like to
thank all the people who have made this thesis possible. Also thanks to the
school for the wonderful research resources and services it provides that help
me through the whole PhD programme.
First and foremost I would like to express my sincerest gratitude to my
supervisor, Prof. Roger Zimmermann, for his persistent guidance and support
throughout my PhD study. His encouragement and patience always help me
overcome the difficulties that I have encountered in life. Thanks for helping
me with paper writing and polishing. Without his inspiration and advices, this
thesis would not have been completed.
IamdeeplygratefultoallmygroupmembersandlabmatesinNUS:Beomjoo
Seo, Yu Yi, Liu Zhenguang, Hao Jia, Shen Zhijie, Ma He, Zhang Ying, Ma
Haiyang, Zhang Lingyan, Cui Weiwei, Rajiv Ratn Shah, Abhay Sharma, Sub-
hasree Basu, Bayan Ta’ani, Abdelhak Betaleb, for their support and accom-
pany. I would also like to thank my friends in Singapore, for the wonderful
years that we have spent together.
Last but not the least, I would like to express my deepest gratitude to my
family. A special love goes to my late father, who had been a great mentor in
my life and had constantly encouraged me to be a better person.

LIST OF PUBLICATIONS
• Yifang Yin, Yi Yu, Roger Zimmermann, “On Generating Content-Oriented
Geo Features for Sensor-Rich Outdoor Video Search”. In IEEE Transactions on
Multimedia, Volume 17 Issue 10, pages 1760-1772, 2015.
• Yifang Yin, Luming Zhang, Roger Zimmermann,“Exploiting Spatial Relation-
shipbetweenScenesforHierarchicalVideoGeotagging”. InProceedingsofACM
International Conference on Multimedia Retrieval, pages 363-370, 2015.
• Yifang Yin, Beomjoo Seo, Roger Zimmermann, “Content vs. Context: Visual
andGeographicInformationUseinVideoLandmarkRetrieval”. InACMTrans-
actions on Multimedia Computing, Communications, and Applications, Volume
11 Issue 2, Article No. 39, pages 39:1-39:21, 2015.
• YifangYin,ZhijieShen,LumingZhang,RogerZimmermann. “Spatial-Temporal
TagMiningforAutomaticGeospatialVideoAnnotation”. InACMTransactions
onMultimediaComputing,Communications,andApplications,Volume11Issue
2, Article No. 29, pages 29:1-29:21, 2014.
• Guanfeng Wang, Yifang Yin, Beomjoo Seo, Roger Zimmermann, Zhijie Shen,
“Orientation data correction with georeferenced mobile videos”. In Proceed-
ings of ACM International Conference on Advances in Geographic Information
Systems, pages 390-393, 2013.
• Guanfeng Wang, Beomjoo Seo, Yifang Yin, Roger Zimmermann, Zhijie Shen,
“OSCOR: an orientation sensor data correction system for mobile generated
contents”. In Proceedings of ACM International Conference on Multimedia,
pages 439-440, 2013.

CONTENTS
Summary v
List of Figures vii
List of Tables xi
1 Introduction 1
1.1 Background and Motivation . . . . . . . . . . . . . . . . . . . . 1
1.2 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.3 Roadmap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2 Literature Review 14
2.1 Geo-metadata Correction . . . . . . . . . . . . . . . . . . . . . . 14
2.2 Geo-tagged Image and Video Annotation . . . . . . . . . . . . . 16
2.3 Landmark Recognition . . . . . . . . . . . . . . . . . . . . . . . 20
2.4 Video Similarity Search . . . . . . . . . . . . . . . . . . . . . . . 23
i
CONTENTS
3 Preliminaries 26
3.1 Viewable Scene Model . . . . . . . . . . . . . . . . . . . . . . . 26
3.2 Automatic Geo-tagged Video Annotation . . . . . . . . . . . . . 28
3.3 Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
3.4 Notations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
4 Automatic Geographic Metadata Correction 33
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
4.2 System Overview . . . . . . . . . . . . . . . . . . . . . . . . . . 36
4.2.1 Design Principles . . . . . . . . . . . . . . . . . . . . . . 36
4.2.2 Problem Description . . . . . . . . . . . . . . . . . . . . 37
4.3 Video Georegistration . . . . . . . . . . . . . . . . . . . . . . . . 38
4.3.1 Camera Model . . . . . . . . . . . . . . . . . . . . . . . 39
4.3.2 Energy Definition . . . . . . . . . . . . . . . . . . . . . . 40
4.3.3 Energy Minimization . . . . . . . . . . . . . . . . . . . . 46
4.4 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
4.4.1 Experimental Setup . . . . . . . . . . . . . . . . . . . . . 47
4.4.2 Geographic Metadata Correction . . . . . . . . . . . . . 48
4.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
5 Spatial-Temporal Tag Mining 55
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
5.2 Review of the Automatic Tag Generation System . . . . . . . . 57
5.2.1 System Overview . . . . . . . . . . . . . . . . . . . . . . 57
5.2.2 Data Source Limitations . . . . . . . . . . . . . . . . . . 57
5.2.3 Seeking More Varied Data Sources . . . . . . . . . . . . 59
5.3 Positioning Social Tags in Spatial-Temporal Domain . . . . . . . 61
ii