Multi-person Tracking by Online Learned Grouping Model with Non-linear Motion Context

14 Pages·2015·5.95 MB·English

IEEETRANSACTIONSONCIRCUITSANDSYSTEMSFORVIDEOTECHNOLOGY, VOL.XX,NO.XX,MONTHYEAR 1
Multi-person Tracking by Online Learned Grouping
Model with Non-linear Motion Context
Xiaojing Chen, Zhen Qin, Le An, Member, IEEE, and Bir Bhanu, Fellow, IEEE
Abstract—An online approach to learn elementary groups
containing only two targets, i.e., pedestrians, for inferring high
53
level context is introduced to improve multi-person tracking. In
most existing data association-based tracking approaches, only
low level information (e.g., time, appearance, and motion) are
used to build the affinity model, and each target is considered
as an independent agent. Unlike those previous methods, in this
paper, an online learned social grouping behavior model is used
to provide more robust tracklets affinities. A disjoint grouping
graph is used to encode social grouping behavior of pairwise
targets, where each node represents an elementary group of two Frame 798 Frame 912
targets, and two nodes are connected if they share a common
target. Probabilities of the uncertain target in two connected
nodes being the same person are inferred from each edge of
the grouping graph. Relationships between elementary groups
are discovered by group tracking, and a non-linear motion
map is used for explaining non-linear motion pattern between
elementary groups. The proposed method is efficient, able to
handle group split and merge, and can be easily integrated into
any basic affinity model. The approach is evaluated on four
datasets, and it shows significant improvements compared with
state-of-the-art methods. Frame 931 Frame 1002
Index Terms—multi-target tracking, data association, social
Fig. 1. Examples in which grouping information is helpful under the
grouping behavior, elementary grouping model.
challenging conditions for tracking in a video. The same color indicates
the same target. Note that for both targets with bounding boxes there are
significant appearance and motion changes due to occlusions and cluttered
I. INTRODUCTION
background.ImagesarefromCAVIARdataset[8]
Automatic tracking of multiple targets simultaneously in
real-world scenes has been an active research topic in com-
putervisionformanyyears,asitiscrucialformanyindustrial consecutive frames that have high probability to contain the
applications and high level analysis, such as visual surveil- sametarget.Thesetrackletsarefurtherlinkedtoproducelong-
lance, human-computer interaction, and anomaly detection. term tracking results. An affinity model integrating multiple
The goal of multi-target tracking is to recover trajectories of visualcues(appearanceandmotioninformation)isformulated
all targets while maintaining consistent identity labels. There tofindthelinkingprobabilitybetweentracklets,andtheglobal
aremanychallengesforthisproblem,suchasilluminationand optimal solution is often obtained by solving the maximum
appearance variation, occlusion, and sudden change in motion a posteriori (MAP) problem using various optimization algo-
[1][2]. As great improvement has been achieved in object rithms.
detection, data association-based tracking (DAT) has become Although much progress has been made in building more
popular recently [3][4][5][6][7]. In the DAT framework, often discriminative appearance and motion models, problems such
apre-learneddetectorisappliedoneachframetoproducede- as identity switch and track fragmentation still exist in cur-
tectionresponsesofalltargets,andshort-termtrackingresults rent association based tracking approaches, especially under
(i.e., tracklets) are generated by associating responses from challenging conditions where appearance or motion of the
target changes abruptly and drastically, as shown in Fig. 1.
X. Chen is with the Department of Computer Science, University of The goal of association optimization is to find the best set
CaliforniaRiverside,Riverside,CA,92521USA,e-mail:[email protected].
of associations with the highest probability for all targets,
Z.QiniswiththeDepartmentofComputerScience,UniversityofCalifor-
niaRiverside,Riverside,CA,92521USA,e-mail:[email protected]. which makes it not necessarily capable of linking each of
L. An is with with the Department of Electrical and Computer Engi- the difficult tracklet pairs. In this paper, we explore high
neering, University of California Riverside, Riverside, 92521 USA, e-mail:
levelcontextualinformation,i.e.,socialgroupingbehavior,for
[email protected].
B.BhanuiswithCenterforResearchinIntelligentSystems,Universityof associating tracklets that are very challenging by using only
CaliforniaRiverside,Riverside,CA,92521USA,e-mail:[email protected]. lower level features (time, appearance, and motion).
Copyright (c) 2015 IEEE. Personal use of this material is permitted.
When there are only a few interactions and occlusions
However, permission to use this material for any other purposes must be
[email protected]. among targets, DAT achieves robust performance. Discrimi-
IEEETRANSACTIONSONCIRCUITSANDSYSTEMSFORVIDEOTECHNOLOGY, VOL.XX,NO.XX,MONTHYEAR 2
The rest of the paper is organized as follows: Section II
Grouping Modeling Tracklet
Tracklets Online Grouping Analysis via Dynamic Graph Association discussesrelatedworkandcontributionsofthispaper;thepro-
• Learning of Elementary Groups
• Group Tracking (non-linear) • Create Virtual Node posed elementary grouping model is described in Section III;
• Graph Inference
experiments are presented in Section IV; and Section V
concludes this paper.
Fig.2. Overviewoftheelementarygroupingmodel.
II. RELATEDWORKANDCONTRIBUTIONS
native descriptors of targets are usually generated using ap- A. Related Work
pearance and motion information from tracklets. Appearance
Visual tracking has attracted extensive research efforts in
model often uses global or part-based color histograms to
recent years, from individual tracker design [12][13] to multi-
match tracklets, and a linear motion model that assumes
tracker fusion [14]. The method proposed in [14] was the
all targets maintain constant speed without motion direction
first work which can jointly exploit both the spatial and the
changeisoftenadoptedtoconstrainmotionsmoothnessoftwo
temporal correlation from multiple tracking results, leading
tracklets. However, these low level descriptors generally fail
to the state-of-the-art tracking performance. Specifically, the
to associate tracklet pairs with long time gap. This is because
temporal information helps identify individual tracker consis-
the appearance of a target might change drastically due to
tency,andthespatialinformationisusedtoestablishpairwise
heavyocclusion,andthelinearmotionmodelisunreliablefor
correlation among multiple trackers.
predicting location of a target after a large time interval.
Traditional filtering-based multi-target tracking methods
Nevertheless, there is often other useful high level contex- process videos on a frame-by-frame basis, which are more
tual information in the scene which can be effectively used suitable for time-critical applications [12][15]. However, such
to mitigate the aforementioned shortcomings. For instance, greedy methods tend to get stuck at a local optimum, with
sociologists have found that up to 70% of pedestrians tend the possible solution space growing exponentially in the
to walk in groups in a crowd, and people in the same group presence of observation gaps. Recently, the focus of multi-
aremorelikelytohavesimilarmotionpatternandbespatially targettrackinghasshiftedtorobustDATschemes,duetotheir
close to each other for better group interaction [9]. Moreover, global reasoning ability of the solution space. With a deferred
pedestrians in the crowd often either consciously or uncon- globalinference,DATismorerobustagainstobservationgaps
sciously follow other individuals with similar destination to resulting from heavy interactions and occlusions [16].
facilitatenavigation[10].Itisalsoobservedinmanyrealworld Huang et al. [13] first propose to hierarchically associate
surveillance videos that if two people are walking together at detection responses for multi-person tracking. Since then,
certain time then it is very likely that these two people will most follow-up works focus on designing features for more
still walk together after a short time period. reliableassociationscoresordevelopingeffectiveoptimization
Basedontheaboveobservations,weproposeanelementary schemes. In the first regime, affinity scores are generally
groupingmodelwithnon-linearmotioncontexttocompensate extracted from appearance information such as color his-
the errors caused by using basic appearance model and linear tograms and motion features such as motion smoothness.
motionmodel.Agroupinggraphisconstructedbasedoninput Globalappearanceconstraintsareexploitedtopreventidentity
tracklets with high confidence, where each node represents a switches in multi-target tracking [17]. Part-based appearance
pairoftrackletsthatformanelementarygroup(agroupoftwo models have been applied in multi-target tracking to mitigate
targets) and each edge indicates that the connected two nodes occlusions [18]. For optimization, bipartite matching via the
(two elementary groups) have at least one target in common. Hungarian algorithm is among the most popular and simplest
The group trajectories of any two linked nodes are used to algorithms [7][13]. A lot of other optimization frameworks
estimate the probability of the other target in each group have been proposed, such as K-shortest path [19], set-cover
being the same person. Neighboring tracklets that have time [20], Linear Programming [21], and Quadratic Boolean Pro-
overlap and similar motion pattern are possible candidates for gramming [22].
elementary groups. Relationships between elementary groups Most of the work only considers pairwise similarities,
are further discovered with the help of group tracking, in without referring to high level contextual information. Thus,
which a non-linear motion map is used to explain large time problems such as possible abrupt motion changes cannot be
gapbetweentwoelementarygroups.Theelementarygrouping properly accounted for. Yang et al. [23] use a Conditional
model is summarized in Fig. 2. Random Field (CRF) for tracking while modeling motion
The size of a group may change dynamically as people dependencies among associated tracklet pairs. Butt et al. [24]
join and leave the group, but a group of any size can always carryoutaLagrangianrelaxationtomakehigher-orderreason-
be considered as a set of elementary groups. Therefore, ing tractable in the min-cost flow framework. These methods
focusingonfindingelementarygroupsinsteadofthecomplete focus on higher-order constraints such as constant velocity.
groupmakesourapproachcapableofmodelingflexiblegroup However, both of them [23][24] concentrate on individuals
evolution [11] in the real world. Note that the social group andmayfailinreal-worldscenarios,inwhichindividualsmay
in this paper refers to a number of individuals with correlated possess a lot of freedom.
movementsanddoesnotindicateagroupofpeoplewhoknow Inthispaper,wefocusonutilizingsocialgroupinginforma-
each other. tion for more natural high-level contextual constraints. Social
IEEETRANSACTIONSONCIRCUITSANDSYSTEMSFORVIDEOTECHNOLOGY, VOL.XX,NO.XX,MONTHYEAR 3
Legend
Confident Tracklets Elementary Groups Grouping Graph
Tracklets
T1 T2 T5 T1-T3 g1 T1-T3 g1 T4-T6 g4 gmTi -Tj EGlreomupe ngtamr y
T3 T4-T6 g4 g2 T6-T7 g5 High Level Info. vm Virtual
T4 T7 T8 T2-T3 g2 T2-T3 T7-T2T8 v2 (grouping behavior) TiTj-Tk Group vm
T6-T7 g5
Matching
T7-T8 g3 T3T7-T8 v1 T7-T8 g3 k Option
T6
Tracking Output
Group
(non-linear)
Tracking
RDeestpeocntisoens T1 T2 T5 g2 2 g3 AsTsroaccikalteiot n
T2-T3 T7-T8

T3 T10
Tracklets
Generation T9 T4 T7 T8
Low Level Info.
(time, appearance, motion)
T6
Fig. 3. Block diagram of our tracking system. After initial tracklets are generated by linking detection responses, confident tracklets are selected to form
elementary groups. The relationships between elementary groups are identified by group tracking with non-linear motion context. Then a disjoint grouping
graph is constructed, from which high level information (i.e., grouping behavior) is extracted. Finally, tracklet association is carried out based on affinity
modelthatcombinesbothhighlevelandlowlevelinformation.Trackletswiththesamecolorcontainthesametarget.Forthelegendsinthisfigureplease
seetheboxintheupperrighthandside.Bestviewedincolor.
factorshaveattractedalotofattentionsinmulti-targettracking enables the proposed method to explain reasonable non-
recently, since they are complementary to unreliable visual linear motions of targets.
featuresandaremotivatedbysociologyresearch.Pellegriniet • The proposed approach based on elementary grouping is
al.[25]proposeamoreeffectivedynamicmodelbyleveraging simple and computationally efficient, while it is effective
nearby people’s positions. Brendel et al. [26] also consider and robust.
nearby tracks as contextual constraints. Alahi et al. [27] • Fourreal-worldsurveillancedatasetsareusedforevalua-
study large-scale crowd destination forecasting with social tionandextensiveexperimentsarecarriedouttovalidate
context.Pellegrinietal.improvetrajectorypredictionaccuracy the effectiveness of the proposed method.
by inferring pedestrian groups [28]. In the DAT context, Apreliminaryversionofthisworkappearedin[32].Inthis
Qin et al. [29] seek the consistency of trajectories in both paper,wehavethefollowingmajorchangesandimprovements
trackletassociationspaceandtrackletgroupassignmentspace as compared to [32]: 1) We study the related work more
based on visual and grouping cues. They use gradient-based extensively and more recent advances are discussed; 2) We
optimization and K-means clustering with multiple random improveourelementarygroupingframeworkbyincorporating
initializations. Bazzani et al. [30] consider joint individual- a modified motion model for group tracking to handle non-
group tracking, with a decentralized particle filter sampling in linear motions of targets; 3) We include more details for
both individual and group spaces. Yan et al. [31] explicitly betterunderstandingofthetechnicalapproach;4)Weconduct
consider group structures to improve tracking consistency more in-depth experiments on more datasets, provide more
across time. Compared to these methods, our approach is comparisons to the state-of-the-art methods.
deterministic with a closed-form solution. Furthermore, the
previous work assumes a static group structure or a fixed III. TECHNICALAPPROACH
number of groups, while our grouping scheme is more flex-
In this section, we introduce how the elementary grouping
ible by using elementary groups and allows for more local
model is integrated into the basic tracking framework for
refinements.
tracklet association. An overview of the proposed method is
presented in Fig. 3.
B. Contributions of This Paper
A. Tracking Framework with Grouping
The contributions of this paper include:
Given a video sequence, a human detector is first applied
• An approach estimating elementary groups online is to each frame to obtain detection responses. Finding the
proposed,whichinfersgroupinginformationtoadjustthe best set of detection associations with the maximum linking
affinity model for data association-based tracking. This probability is the aim of detection-based tracking. In an
approach is independent of detection methods, affinity ideal association, each disjoint string of detections should
models, and optimization algorithms. correspond to the trajectory of a specific target in the ground-
• A motion model that takes advantage of nearby non- truth. However, object detector is prone to errors, such as
linearmotionpatternsisintegratedintogrouptracking.It false alarms and inaccurate detections. Also, directly linking
IEEETRANSACTIONSONCIRCUITSANDSYSTEMSFORVIDEOTECHNOLOGY, VOL.XX,NO.XX,MONTHYEAR 4
detectionsincurahighcomputationalcost.Inordertogenerate Therefore,weproposeelementarygroupsthatarelearnedand
asetofreliabletracklets(trajectoryfragments),therefore,itis updated“online”,duringthetrackingprocesstoprovideuseful
a common practice to pre-link detection responses that have social grouping information while maintaining the flexibility
high probability to contain the same person. Next, a global of the group structure. Two tracklets T and T are likely to
i j
optimizationmethodisemployedtoassociatetrackletsaccord- correspond to the same target if they satisfy the following
ingtomultiplecues.Finally,misseddetectionsareinsertedby constraints: 1) each of them forms an elementary group with
interpolation between the linked tracklets. Detections that do the same tracklet, namely, the same target; 2) the trajectory
not belong to any tracklet or tracklets that are too short are obtainedbylinkingT andT hasasmalldistancetothegroup
i j
considered as false alarms and removed from the final results. meantrajectory.Thefirstconstraintisbasedontheobservation
Amathematicalformulationofthetrackingproblemisgiven that if two people are walking together for a certain time,
as follows. Suppose a set of tracklets T = {T ,..,T } is then there is high probability that they will still walk together
1 n
generatedfromavideosequence.AtrackletT isaconsecutive after a short time period. The second constraint prevents us
i
sequence of detection responses or interpolated responses that from linking wrong pair of tracklets. Let P be the inferred
ij
contain the same target. The goal is to associate tracklets that high level information for T and T , the tracklet association
i j
correspond to the same target, given certain spatial-temporal problem can be refined as:
constraints. Let association a defines the hypothesis that
ij (cid:88)
trackletT andT containthesametarget,assumingT occurrs A∗ =argmin a (S −αP ), (3)
i j i ij ij ij
before Tj. A valid association matrix A is defined as follows: A∈Ω ij
where α is a weighting parameter. It is selected by coarse
(cid:26)
1, if T is associated to T , binary search in only one time window and kept fixed for all
A={a },a = i j (1)
ij ij 0, otherwise, the others.
(cid:88)n (cid:88)n In the following, we introduce an online method for group
s.t. a =1 and a =1.
ij ij analysis and obtain P by making inferences from the group-
i=1 j=1 ij
ing graph.
TheconstraintsformatrixAindicatethateachtrackletshould
beassociatedtoandassociatedbyonlyoneothertracklet(the
initial and the terminating tracklets of a track are discussed in B. Learning of the Elementary Groups
Section IV-A). In this part, we explain how the nodes (elementary groups)
WedefineSij asthebasiccostforlinkingtrackletTi andTj of the grouping graph are created. A set of tracklets is gen-
basedonlowlevelinformation(time,appearance,andmotion). erated after low level association, but only confident tracklets
It is computed as the negative log-likelihood of Ti and Tj are considered for grouping analysis, as there might be false
being the same target (explained in detail in Section IV-A). alarms which may lead to incorrect associations in the input
Note that Sij =∞ if Ti and Tj have overlap in time. tracklets. Based on the observation that inaccurate tracklets
Let Ω be the set of all possible association matrices, are often the short ones, we define a tracklet as confident if it
the multi-target tracking can be formulated as the following is long enough (e.g., it exists for at least 10 frames).
optimization problem: Two tracklets T and T form an elementary group if they
i j
have the following properties: 1) T and T have overlap in
(cid:88) i j
A∗ =argmin aijSij. (2) time for more than l frames (l is set to 5 in our experiments);
A∈Ω ij 2) they are spatially close to each other; 3) they have similar
This assignment problem can be solved optimally by the velocities.Mathematically,weuseGij todenotetheprobabil-
Hungarian algorithm in polynomial time. In order to reduce ity of Ti and Tj forming an elementary group:
computational cost, the video is segmented by a pre-defined
time sliding window, which is fixed to be 12 seconds long. Gij =Pt(Ti,Tj)·Pd(Ti,Tj)·Pv(Ti,Tj), (4)
Trackletassociationiscarriedoutineachtimeslidingwindow.
where P (·), P (·) and P (·) are the grouping probabilities
t d v
There has to be a 50% overlap between two neighboring time
based on overlap in time, distance and velocity, respectively.
windows. To handle association conflicts in the overlapping
Their definitions are given in Eq. (5), Eq. (6), Eq. (7).
part of two windows, we use a method similar to [23]. More
specifically, the overlapped part is evenly divided into two P (T ,T )= Lij , (5)
parts.Inthefirsthalf,trackingresultsproducedbytheprevious t i j L +l
ij
time window is kept, while in the second half, original input
tracklets are used despite the association results from the P (T ,T )= 1 (cid:88)Lij (1− 2 arctan(dist )), (6)
previous time window. d i j Lij n=1 π n
As low level information is not sufficient to distinguish
cosθ+1
targets under challenging situations, we consider to integrate
P (T ,T )= , (7)
high-level information from social grouping behavior into v i j 2
the cost matrix to regularize the solution. However, group where L is the length of overlapped frames for T and T ,
ij i j
configuration is often not known a priori. Also, it is not dist is the normalized center distance for T and T on the
n i j
fixed for the entire video, as people might change groups. nth overlapped frame, and θ is the angle between the average
IEEETRANSACTIONSONCIRCUITSANDSYSTEMSFORVIDEOTECHNOLOGY, VOL.XX,NO.XX,MONTHYEAR 5
motion, respectively. Similar to Eq. (2), let Φ be the set of all
13
12
11 16 possible group association matrices, then the group tracking
11 can be formulated as the following optimization problem:
(cid:88)
Ag∗ =argmin a Cg(g ,g ). (9)
ij i j
Ag∈Φ
ij
Hungarianalgorithmisusedtosolvethisassignmentproblem.
(a) (b) 1) Time Model for Group Tracking: For the linking cost
based on time, we defined it as:
Fig.4. Examplesofgeneratingincorrectelementarygroupsifthedistances
arenotnormalized. (cid:26)
0, g is not overlapped with g ,
Cg(g ,g )= i j (10)
t i j ∞, otherwise,
velocities of the two tracklets during the overlapped frames.
In our experiments, distn is set as follows: where the non-overlapping constraint means any tracklet in gi
has no time overlap with any tracklet in g .
j
distn =ration·d/0.5(widthi+widthj), (8) twoIfmgaitcahnidnggjpocsosnibtailiintietsh:e1)saTmgei atnwdoTtagjrgaertes,ththeesraemaeretarognelty,
1 1
where ration is the size of the larger target over the size of T2gi andT2gj arethesametarget;2)T1gi andT2gj arethesame
the smaller target, d is the Euclidean distance between the target, Tgi and Tgj are the same target. We explain in detail
2 1
two object centers, and 0.5(widthi+widthj) is the smallest formatchingoption1),notethatthecomputationformatching
distance in the image space for two people that walk side by option2)issimilar.Foreachmatchingoption,wecomputethe
side. The term ration prevents tracklets as shown in Fig. 4 linkingcostbasedonappearanceandmotion,andusetheone
to be considered as a group, where the distance in the image with the smaller sum for Cg (g ,g )+Cg (g ,g ). Also,
appr i j mt i j
spaceissmallwhilethedistanceinthe3Dspaceisquitelarge. the matching option is recorded for each group association.
We create a node for each pair of tracklets that have non- 2) Appearance Model for Group Tracking: Let S(·) be the
zero grouping probability G. Thus, each node contains two appearance similarity for two tracklets, the group linking cost
tracklets/targets and is associated with a probability G, its based on appearance is defined as:
value indicates the similarity of motion patterns for these two
tracklets during their co-existence period.
Note that if two tracklets form an elementary group, their Cg (g ,g )=−ln(0.5(S(Tgi,Tgj)+S(Tgi,Tgj))). (11)
appr i j 1 1 2 2
group mean trajectory is obtained by computing the mean
position using only their overlapping parts, as the grouping is As there might be appearance variations in a single tracklet
only meaningful for the overlapped time period. For example, due to occlusion and lighting changes, it is hard to generate
if T and T are in the same elementary group, this only features that can robustly represent the appearance of a target.
a b
indicates that T and T have similar motion patterns for In order to more reliably compute the similarity between two
a b
the period that they have time overlap. During the non- tracklets, we adopt the modified Hausdorff metric [33] which
overlapping period, T may form elementary groups with is able to compute the similarity of two sets of images. Given
a
other tracklets/targets that are even in a different group than a tracklet Ti that has length mi, let Ti = {di1,di2,...,dimi}
the group of Tb. Such property makes the elementary group where dix is the xth estimation of Ti, then S(·) is defined as:
flexible to handle group split and merge.
1 (cid:88) 1 (cid:88)
S(T ,T )=min( s(di,T ), s(dj,T )),
C. Group Tracking i j mi x j mj y i
dix∈Ti djy∈Tj
The relationship between two elementary groups is identi- (12)
fied by group tracking. Inspired by association-based multi- where s(d,T) = max (s (d,d(cid:48))) is the Hausdorff sim-
d(cid:48)∈T cos
target tracking, we define our group tracking as a problem ilarity between an estimation and a tracklet. A modified
of finding globally optimal associations between elementary cosine similarity measure [34] s (·) is used to compute the
cos
groups based on the three most commonly used features: similarity between two estimations, which is defined as
time, appearance, and motion. More specifically, given a set
(cid:12) (cid:12)
of elementary groups, we compute the linking cost for any (cid:12)uT ·v(cid:12)
s (u,v)= , (13)
two groups and obtain the association results by finding the cos (cid:107)u(cid:107)(cid:107)v(cid:107)((cid:107)u−v(cid:107) +(cid:15))
p
association set with the minimum total cost.
Let {Tgi,Tgi} denote the two tracklets in an elementary whereu,varethefeaturedescriptorsfromtwoimages,(cid:107)·(cid:107) is
1 2 p
groupg .Giventwoelementarygroupsg andg ,assumingg the l norm (we set p=2), and (cid:15) is a small positive number
i i j i p
startsbeforeg ,theirlinkingcostisCg(g ,g )=Cg(g ,g )+ to avoid dividing by zero. In our experiments, we use the
j i j t i j
Cg (g ,g ) + Cg (g ,g ), where Cg(·), Cg (·), and concatenation of HSV color histogram and HOG features as
appr i j mt i j t appr
Cg (·) are linking costs based on time, appearance, and the feature descriptors.
mt
T1
T2 T5
T3
T8
T4 T7
T6
IEEETRANSACTIONSONCIRCUITSANDSYSTEMSFORVIDEOTECHNOLOGY, VOL.XX,NO.XX,MONTHYEAR 6
T is a matched tracklet for it. The motion cost for linking
x
g and g based on non-linear prediction of locations can be
a b
g computed according to Eq. (14).
b
For each pair of elementary groups, both linear and non-
a matched
group mean linear motion models are used, and the score with a lower
non-linear estimated
trajectory cost is selected. Note that when only linear motion model
motion pattern path
is used, any trajectory not following the pattern is penalized.
g Withthenon-linearmotionmodel,weareabletoexplainnon-
a
Tx linear motion in the scene without producing extra penalties
for individuals who do not follow a linear motion pattern.
D. Creation of Virtual Nodes
Fig.5. Anexampleofestimatingmotionaffinityusingthenon-linearmotion
map. Our goal is to encode grouping structure of the tracklets
by the elementary grouping graph. With elementary groups
as nodes of the graph, we define an edge between two nodes
3) Motion Model for Group Tracking: We measure the
indicating the existence of at least one common target in the
motion affinity of two elementary groups by the motion
correspondingtwoelementarygroups.Forsimplecaseswhere
smoothness between the group mean trajectories of the two
two nodes have one tracklet in common, we link these two
correspondingelementarygroups.Themotioncostforlinking
nodes directly, such as nodes g and g , g and g shown
two group mean trajectories is defined as the negative loga- 1 2 4 5
in Fig. 3. For difficult cases where there are four different
rithm of the motion affinity:
tracklets in two nodes, we use the results of group tracking to
find their relationship.
Cg (g ,g )=−ln(G(f (g ,+∆t)−pgj ,Σ ) (14) Supposeg andg areassociatedbygrouptracking,namely,
mt i j predict i head p i j
·G(f (g ,−∆t)−pgi ,Σ )), these two elementary groups contain the same two targets.
predict j tail p
We create two virtual nodes v and v , set their grouping
p q
whereG(·)isazeromeanGaussiandistribution,∆tisthetime probability G to be the same as that of node g , and build
j
gap between gi and gj, fpredict(gi,±∆t) gives the location edges between gi and the virtual nodes. Note that the virtual
prediction for the group mean trajectory of gi after (+) or nodes can also be added in the other way (i.e., set G to be
before (−) ∆t, phead and ptail are the head and tail locations the same as gi and link the virtual nodes to gj), but these two
for a group mean trajectory. options are exclusive to each other. Each virtual node also
In most previous tracking frameworks [2][29][35], tar- contains two tracklets, one is a virtual tracklet generated by
gets are commonly assumed to maintain linear motion linking a pair of matched tracklets in g and g , the other is
i j
pattern. Thus, fpredict(gi,+∆t) = pgtaiil + vtgaiil∆t and the tracklet left in gj. An example of virtual node creation is
fpredict(gj,−∆t)=pghjead−vhgjead∆t. However, in real world presentedinFig.3.Basedontheassociationofg2 andg3,two
scenarios, it is common to observe several non-linear motion virtual nodes v and v are created and connected to g . Two
1 2 2
patterns in the scene. In order to produce more robust motion virtual nodes are used since there are two pairs of tracklets
affinity for two elementary groups, we use the non-linear thatneedinference(edgeforg andv indicatesinferencefor
2 1
motion map [5] to explain large non-linear time gaps between T and T ; edge for g and v indicates inference for T and
2 8 2 2 3
groupmeantrajectories.Notethatin[5]thenon-linearmotion T ). In the following, we show that by using the virtual node
7
map is directly used to estimate the motion affinity of two inference can be easily done.
tracklets, whereas we use it for explaining non-linear gap
between two elementary groups.
The non-linear motion map M is a set of all existing non- E. Inference from the Grouping Graph
linear tracklets in current time sliding window, and the track- Inthegroupinggraph,eachnodeisanelementarygroupand
lets are selected only from the confident ones. An example each edge indicates that the two connected elementary groups
of estimating motion affinity between ga and gb using a non- have one target in common. According to the observation that
linear motion pattern Tx in the motion map is illustrated in two people walk together at certain time are likely to walk
Fig. 5. The tracklet Tx ∈ M is a non-linear motion pattern together after a short period, given two directly connected
that has co-existed in time with both ga and gb and is a groups, we can infer the probability of the uncertain target
matched tracklet for the group mean trajectories of ga and in each group to be the same.
gb. Tx is a matched tracklet indicates that it is spatially close Suppose there is an edge between nodes gi and gj in the
to the elementary group and has similar motion direction as grouping graph, assuming Ti = Tj = T , Ti = T , and
1 1 k 2 l
the elementary group. Then a quadratic curve that best fits Tj =T without loss of generality, the probability of Ti and
2 m 2
positions at the tail part of ga and the head part of gb is Tj contain the same target is defined as follows:
2
estimated to fill the path between g and g . Therefore, each
a b
groupassociationhasaspecificquadraticfunctionforitsnon-
linear motion estimation. The estimated path is only valid if p =0.5(G +G )×TSimi(T ,G ), (15)
lm kl km {l,m} {k,l,m}
IEEETRANSACTIONSONCIRCUITSANDSYSTEMSFORVIDEOTECHNOLOGY, VOL.XX,NO.XX,MONTHYEAR 7
shown in Algorithm 1.
T1 T1
T2 T5 T2 T5
Algorithm 1 Learning algorithm for elementary grouping model
T3 T3
T8 T8 Input: Tracklet set T ={T ,..,T }
T4 T4 1 n
T7 T7 Output: Inference matrix P, where P is the inference for
ij
T6 T6 Ti and Tj
1: P←empty set, Nodes←∅, Edges←∅
(a) (b)
2: for i=1,...,n do
T1 T1 3: for j =i+1,...,n do
T2 T5 T2 T5
4: if Ti and Tj are confident tracklets then
T3 T3 5: Gij =Pt(Ti,Tj)Pd(Ti,Tj)Pv(Ti,Tj)
T4 T8 T4 T7 T8 6: if Gij >0 then
T7
7: Create node g ={Ti,Tj}
T6 T6 8: Nodes=Nodes∪{g}
(c) (d) 9: for i=1,...,size(Nodes) do
10: for j =i+1,...,size(Nodes) do
Fig. 6. Inference for each edge in the grouping graph in Fig. 3: (a) edge 11: if ∃T ∈gi, T =T1gj or T =T2gj then
(brdeep)twreedesgeenentbgse1itnwatneedrepngog2la4,t(iabon)ndebdgeg5tew(sbeeeeentwFteihgee.n3tgwf2ooratngrdarocvuk1ple,at(sncn)thoeatdatgtnieoenbesed)t.wiBnefleeanrcekgn2scoeal,nidbdllavicn2ke, 1123:: CEdregaetse=anEeddggees∪e{{gie,{ggj}i,gfjo}r}gi and gj
dashedlineisthegroupmeantrajectory,andcoloreddottedlineindicatesa 14: Update Nodes and Edges according to group tracking
virtualtracklet.Bestviewedincolor. 15: for all e∈Edges do
16: Computepxy forthecorrespondingtrackletpairusing
Input Tracklets Grouping Graph
Eq. (15)
T1 T1-T2 g1 T2-T4 g3 17: Update P: Pxy =Pxy+pxy
T2
T3
T4
T1-T3 g2 T3-T4 g4 IV. EXPERIMENTS
We evaluate our approach on four datasets: the CAVIAR
dataset[8],theTownCentredataset[35],thePETS2009dataset
Fig.7. Anexampleofmultipleinferencesrelatedtothesametwotracklets. [36], and the UNIV dataset [37]. The popular evaluation
According to the proposed elementary grouping model, a grouping graph
metrics defined in [38] and the CLEAR MOT metrics defined
(shown on the right) is created based on the input tracklets (shown on the
left). Thus, inferences based on the edge between node g1 and g2 and the in [39] are used for performance comparison:
edgebetweennodeg3 andg4 areallrelatedtotrackletsT2 andT3. - GT the number of trajectories in the ground-truth.
- MT the ratio of mostly tracked trajectories, which are
successfully tracked for more than 80% of the time.
where TSimi(T ,G ) is the trajectory similarity
{l,m} {k,l,m}
- ML the ratio of mostly lost trajectories, which are suc-
between trajectory T (created by linking T and T ) and
{l,m} l m
cessfully tracked for less than 20% of the time.
thegroupmeantrajectoryG (createdbycomputingthe
{k,l,m}
- Frag fragments, the number of times that a ground-truth
mean position of T and T ). We define the trajectory
k {l,m}
trajectory is interrupted.
similarity as follows:
- IDS ID switches, the number of times that a tracked
2 trajectory changes its matched id.
TSimi(T,G)=1− arctan(Dist), (16)
π - FP false positive, the number of tracker hypotheses for
which no real object exists.
where Dist is the average Euclidean distance of trajectory T
- FN false negative, the number of times that targets have
and group mean trajectory G.
no matched hypothesis.
Foredgesconnectingtwonormalnodesandedgesconnect-
-MOTAmultipleobjecttrackingaccuracy,acombinedmea-
ing to one virtual node, the same inference function can be
sure which takes into account false positives, false negatives
used. The only difference is that the latter uses one virtual
and identity switches.
trackletandtwonormaltrackletsasinput.Examplesofmaking
- MOTP multiple object tracking precision, measures the
inference for a grouping graph are shown in Fig. 6. Note that
alignment of tracking results with respect to ground-truth.
there might be multiple inferences related to the same two
The following tracking approaches are tested:
tracklets, as the same tracklet may be contained in multiple
elementary groups, as shown in Fig. 7. Therefore, P in Eq. • Our Model (non-linear): the proposed elementary group-
ij
(3) is the sum of all inferences that relate to T and T : ing model with non-linear motion context for group
i j
tracking.
P =(cid:88)p . (17) • Our Model (linear): the proposed elementary grouping
ij ij
model with only linear motion model for group tracking.
A summary of the proposed elementary grouping model is • Baseline Model 1: the basic affinity model.
IEEETRANSACTIONSONCIRCUITSANDSYSTEMSFORVIDEOTECHNOLOGY, VOL.XX,NO.XX,MONTHYEAR 8
TABLEI
COMPARISONOFTRACKINGRESULTSONCAVIARDATASET.THENUMBEROFTRAJECTORIESINTHEGROUND-TRUTH(GT)IS75.
Method MT ML Frag IDS FP FN MOTA MOTP Time
BaselineModel1 74.7% 6.7% 11 12 1459 10827 79.2% 78.8% 1.5s
BaselineModel2 78.7% 6.7% 10 8 1535 9134 82.0% 81.7% 4.2s
SGBModel[29] 89.3% 2.7% 7 5 1597 8497 83.0% 82.1% 50s
OurModel(linear) 90.7% 2.7% 6 5 1668 8081 83.5% 82.0% 4.6s
OurModel(non-linear) 90.7% 2.7% 6 5 1668 8081 83.5% 82.0% 6.1s
TABLEII
COMPARISONOFTRACKINGRESULTSONTOWNCENTREDATASET.THENUMBEROFTRAJECTORIESINTHEGROUND-TRUTH(GT)IS220.
Method MT ML Frag IDS FP FN MOTA MOTP Time
BaselineModel1 76.8% 7.7% 37 60 2746 28493 56.1% 68.8% 350s
BaselineModel2 78.6% 6.8% 34 46 3155 22236 64.3% 71.3% 457s
SGBModel[29] 83.2% 5.9% 28 39 4387 15871 81.8% 69.7% 4861s
OurModel(linear) 85.5% 5.9% 26 36 4105 14804 73.4% 69.2% 465s
OurModel(non-linear) 86.4% 5.9% 25 36 4938 13910 73.5% 69.2% 505s
• Baseline Model 2: the proposed elementary grouping are adopted: time, appearance and motion. The basic affinity
model without group tracking. P for two tracklets T and T is defined as
basic i j
• SGB: the Social Grouping Behavior model [29].
For a fair comparison, the same input tracklet set, ground-
P (T ,T )=f (T ,T )·f (T ,T )·f (T ,T ). (18)
basic i j t i j appr i j mt i j
truth, as well as basic affinity model are used for all the
methods.AlltheresultsfortheSGBmodelarekindlyprovided Thetimeaffinitymodelftassignszeroaffinitytotrackletpairs
bytheauthorsof[29].Bothquantitativecomparisonswiththe whose time gap is greater than a pre-defined threshold GAP,
state-of-the-artmethodsandvisualresultsofourapproachare it is defined as
presented.
(cid:26)
0, if Gap >GAP,
f (T ,T )= ij (19)
t i j 1, otherwise.
A. Implementation Details
The appearance affinity model f is based on the Bhat-
Tracklets generation: Two different ways of generating appr
tacharyya coefficient of two average HSV color histograms.
trackletsareemployedtodemonstratethattheproposedgroup-
Forthemotionaffinitymodelf ,thesamemethodasshown
ingmodelcanbeeasilyintegratedintoanyDATbasedtracking mt
in Eq. (14) with linear motion for f is used to measure
system, regardless of the method used to extract the initial predict
the motion smoothness of two tracklets in both forward and
tracklets.Inthefirstmethod,targetsoneachframearedetected
backward directions. Given P (T ,T ), the basic cost S
usingthediscriminativelytraineddeformablepart-basedmod- basic i j ij
in Eq. (2) is computed as S =−ln(P (T ,T )).
els [40]. We apply a nearest neighbor detection association ij basic i j
Cost matrix S: Due to the constraints in Eq. (1), the
method similar to [7] to generate the initial tracklets. For
traditionalpairwiseassignmentalgorithmisnotabletofindthe
each unassociated detection a Kalman filter based tracker is
initialandtheterminatingtracklets.Therefore,insteadofusing
initialized with position and velocity states. A detection A is
thecostmatrixS(n×n)directly,weusetheaugmentedmatrix
associated to a detection B in the next frame if B has the
(2n × 2n) proposed in [29] as the input for the Hungarian
minimum distance to the predicted location and overlaps at
algorithm. This enables us to set a threshold for association,
least50%(measuredassize(A∩B)/size(A∪B))insizewith
a pair of tracklets can only be associated when their cost is
detection A. Then the corresponding Kalman filter is updated
lower than the threshold. In our experiments, the threshold is
with the newly associated detection. The tracker terminates if
set to −ln0.5 for all datasets.
noassociationisfoundformorethantwoconsecutiveframes,
or a detection is associated by multiple trackers.
B. Results on CAVIAR Dataset
In the second method, the popular HOG based human
detector [41] is used. Tracklets are generated by connecting The videos in the CAVIAR dataset are acquired in a
detections in consecutive frames that have high similarity in shopping center where frequent interactions and occlusions
appearance and have large overlap in size. A simple two- occur and people are more likely to walk in groups. We
threshold strategy [13] is used to generate reliable tracklets. select the same set of test videos as in [29], which are the
In our experiments, two detections are connected if and only relatively challenging ones in the dataset. We generate input
if: 1) their affinity is higher than 90%; 2) their affinity is at trackletsusingthefirstmethoddescribedinSectionIV-A.The
least 20% larger than the affinities of any other alternatives. comparativeresultsareshowninTableI.Ourproposedmodels
Basic affinity model: In order to produce reasonable basic (both linear and non-linear) achieve the best overall tracking
affinity for a pair of tracklets, three commonly used features accuracy(MOTA)withthehightrackingprecision(MOTP)as
IEEETRANSACTIONSONCIRCUITSANDSYSTEMSFORVIDEOTECHNOLOGY, VOL.XX,NO.XX,MONTHYEAR 9
TABLEIII
COMPARISONOFTRACKINGRESULTSONPETS2009DATASET.THENUMBEROFTRAJECTORIESINTHEGROUND-TRUTH(GT)IS74.
Method MT ML Frag IDS FP FN MOTA MOTP Time
BaselineModel1 14.9% 64.9% 120 88 271 5414 32.4% 60.5% 297s
BaselineModel2 21.6% 50% 104 102 436 4773 37.8% 59.7% 381s
SGBModel[29] 23% 41.9% 95 91 691 3828 46.0% 59.9% 4962s
OurModel(linear) 28.4% 44.6% 93 97 683 3987 44.1% 58.8% 477s
OurModel(non-linear) 33.8% 35.1% 79 89 729 3081 54.3% 60.1% 612s
TABLEIV
COMPARISONOFTRACKINGRESULTSONUNIVDATASET.THENUMBEROFTRAJECTORIESINTHEGROUND-TRUTH(GT)IS40.
Method MT ML Frag IDS FP FN MOTA MOTP Time
SGBModel[29] 75% 5% 38 7 213 443 96.7% 82.9% 47s
OurModel(linear) 87.5% 5% 26 5 224 287 97.4% 83.1% 3.9s
OurModel(non-linear) 87.5% 5% 26 5 224 287 97.4% 83.1% 4.2s
comparedtotheotheralternatives.Itisobservedthatthebasic to 33 targets per frame) and includes many non-linear motion
affinity model (Baseline Model 1) can produce reasonable patterns.Arectangularareaisdefinedintheworldcoordinates
tracking results, and the performance is further improved by and used as the boundary of the tracking area (as shown in
integrating high-level grouping information (Baseline Model Fig. 10), trajectories outside the area are excluded from our
2, Our Model (linear), and Our Model (non-linear)). Both solution. The first method described in Section IV-A is used
linear and non-linear versions of our model have comparable to generate input tracklets. The comparative results are shown
or better performances in most metrics as compared to the in Table III. We can see that when many non-linear walking
SGB model (e.g., better results in MT and Frag, the same patterns present in the dataset, significant improvements are
results in ML and IDS), but with much less computational achieved by integrating non-linear motion context into the
time. The comparisons between Baseline Model 2 and Our tracking system. Our model with non-linear context gives the
Model(bothlinearandnon-linear)demonstratetheimportance best MOTA and has a higher MT (33.8%) and a lower ML
of group tracking, as they reveal more grouping information. (35.1%)comparedtotheSGBmethod(MT:23%,ML:41.9%)
Sincemostpedestriansinthevideosarewalkinglinearlyalong and Our Model (linear) (MT: 28.4%, ML: 44.6%) that only
acorridorinthisdataset,thereisbarelyanynon-linearcontext consider linear motion during grouping. Also the number of
in the scene. Therefore, the linear and non-linear versions of fragments and ID switches are greatly reduced when social
our model have the same performance (except computational grouping and non-linear context are employed in the tracking
time) on this dataset. Sample tracking results are shown in system. Sample tracking results of the proposed method with
Fig. 8. non-linear motion context are shown in Fig. 10. In the first
row of Fig. 12 we present tracking examples of our method
with linear motion model on the same sample sequence as
C. Results on TownCentre Dataset
shown in Fig. 10.
The TownCentre dataset has one high-resolution video
which captures the scene of a busy street. There are 220
E. Results on UNIV Dataset
people in total, with an average of 16 people visible per
Tofurtherevaluatetheeffectivenessoftheproposedmethod
frame. We test all models using the first 3 minutes of the
in handling dynamics of social groups (e.g., group merge and
video, and generate input tracklets using the second method
split), four video sequences are collected from an elevated
described in Section IV-A. The comparative results are shown
viewpoint that allows the capture of rich group evolving
in Table II. Similar to the observations from Table I, Table II
scenarios.Eachvideoisabout30secondslongwithanaverage
suggests that the performance of our method is consistent on
of 9 pedestrians visible in each frame, some sample frames
both datasets. As there are some non-linear motion in this
are shown in Fig. 11. The input tracklets for this dataset are
dataset, the tracking performance is slightly improved by the
produced using the second method described in Section IV-A.
incorporation of non-linear context. Sample tracking results
Multi-target tracking is carried out using only the grouping
are shown in Fig. 9.
information, namely, the linking costs for tracklet pairs are
based only on P in Eq. (3). The comparative results are
ij
D. Results on PETS2009 Dataset
shown in Table IV. Our model with both linear and non-
We select sequence S2L2 in the PETS2009 dataset to eval- linear motion have the same performance, as this dataset
uate the performance of the proposed method. This sequence contains little non-linear motion pattern. Compared to the
captures the outdoor scene of a campus from an elevated SGB model which assumes a fixed number of groups in the
viewpoint. Unlike the widely used sequence S2L1, sequence scene, our grouping model improves MT by 12.5%, reduces
S2L2 is more challenging as it has higher crowd density (up the fragments by 31.5%, and also achieves higher MOTA and
IEEETRANSACTIONSONCIRCUITSANDSYSTEMSFORVIDEOTECHNOLOGY, VOL.XX,NO.XX,MONTHYEAR 10
2 3 2 3 2 3 2 3
4 5 6 4 56 46 5 6 4 5
1 7 8 7 8
7 8 1 7 8
Frame 200 Frame 220 Frame 260 Frame 290
(a) Track targets (4, 5, 6) when appearances vary a lot due to occlusions
7 8 7 8 7 8 7 8
122 10 31 1 13 122 10 3 111 3 122 101 31 13 12 21 0 11 133
Frame 860 Frame 910 Frame 960 Frame 1000
(b) Successfully tracking targets (11, 13) with long time gap
7 7 7 7
2 2
2 2
5
3 4 6 5 3 64 5 3 6 4 5 6 4
3
Frame 430 Frame 460 Frame 470 Frame 510
(c) Track targets (4, 5, 6) when sudden motion change and occlusion happen
Fig.8. ExamplesoftrackingresultsofourapproachonCAVIARdataset.Thesamecolorindicatesthesametarget.Bestviewedincolor.
209 167 194 1651 60 209 1651 60 212 209 1651 60 209 213
20119 9 171217 7571 791 85 117810178 7 26 166 21019179 211 77571 791 85 1811778 1027 6 1661 94 172121 707151179 7 99 185 18178 127 107 6 116964 179 21018195 9 18178 2 17107 6 183 19146 6
171 203 183
181 183 171
183 181 171
181
Frame 3380 Frame 3400 Frame 3420 Frame 3460
Fig.9. ExamplesoftrackingresultsofourapproachonTownCentredataset.Withgroupinginformation,targets(199and201)pointedbyarrowsarecorrectly
trackedunderfrequentocclusions.Thesamecolorindicatesthesametarget.Bestviewedincolor.