Point trajectories are sparse on low textured regions, e.g., the road in Figure 3.2. Lack of texture results in ambiguous motion estimates. Furthermore, trajectories on untextured backgrounds are often “dragged” by nearby occluding boundaries, a phenomenon referred as optical flow “bleeding” inThompson(1998).
While lack of texture causes ambiguity in optical flow estimation, at the same time, untextured regions usually have salient boundaries, easy to detect from appearance cues. We cast mapping of trajectory clusters to pixel regions as a seeded superpixel partitioning problem. Seeds are provided from superpixels that well overlap with trajectory clusters. We propagate seed labels to non-seed superpixels via random walkers on multiscale su- perpixel affinity graphs. We show such mapping is efficient and robust to low image texturedness and optical flow bleeding. Details are presented right below.
Spatio-temporal multiscale region graphs
Given a video sequenceI, we compute a set of superpixels by thresholding the output of globalPb ofArbelaez et al.(2009) at valueβminat each frame. LetR={rp, p= 1· · ·nR}
notationrp to denote both the pth superpixel as well as its pixel mask. Lettp denote the
frame of superpixelrp.
While superpixels rarely leak across object boundaries, their spatial support is often too small to compute a reliable mapping with trajectory clusters: many of the superpixels overlap with trajectory gaps, as shown in Figure 3.5. We will use random walkers on region affinity graphs to propagate labels of well regions well overlapping with trajectory clusters (seeds) to ambiguous ones.
We establish intra-frame and cross-frame superpixel affinitiesAR ∈RnR×nR. In each
frame, we compute multi-scale superpixel affinities from ultra-contour maps ofArbelaez et al.(2009). The ultra-contour map provides a different superpixel labelingst,β ∈Nnt
R×1
for each probability of boundary thresholdβ ∈[βmin, 1], wherentRthe number of super-
pixels at framet. The intra-frame superpixel affinities are as follows:
AR(rp,rq) = max β,stp,βp =s tq ,β q exp(− β 3 0.12)·δ(tp =tq). (3.10)
intuitively, the affinity between two superpixels of the same frametdepends on the thresh- oldβ for which they have the same label inst,β, the higherβ the lower the affinity. In this
way, intra-frame superpixel affinities isARhave large spatial connection radius. In each
frame, they do not form a planar graph as is often the case in the literature, where each su- perpixel is connected only to its spatial neighbors. Such long range connectivity between superpixels is lost once globalPb is thresholded at a single scale and resulting regions are treated as independent.
We compute cross-frame superpixel affinities from optical flow. Let r+
p denote the
pixel mask after translating pixels inrpwith their optical flow displacements:r+p ={(xp+
up, yp+vp),p∈rp}. The cross-frame superpixel affinities are as follows:
AR(rp,rq) =
|r+p ∩rq| |r+
p ∪rq|·
δ(tp =tq+ 1). (3.11)
Cross-frame region affinities are established only between regions of consecutive video frames. Affinities between regions of non-adjacent frames can be considered using point trajectory overlap, as inGalasso et al.(2012).
intra-frame affinities from
multiscale Pb cross-frame affinities from optical flow
trajectory clustering Delaunay labelling seed / non-seed regions
Random walker
A
RxM
Figure 3.5: Random walkers on spatio-temporal region graphs. The region graph AR
extends across multiple frames.
Trajectory seeded superpixel labeling
Letl ∈ LnT×1 denote a trajectory labeling, whereL={1· · ·L}is the trajectory label set.
We want to estimate a corresponding superpixel labeling.
We partition superpixels into seeds (marked) and non seeds (unmarked) depending on their overlap with labeled trajectory Delaunay triangles, as shown in Figure 3.5. We assign to each Delaunay triangle the label shared by its vertices or leave it unlabeled if its vertices do not have the same label. We then compute intersection of each superpixelr with the colored Delaunay triangulation. Seed regions are those that have more that50% overlap with a trajectory label. We want to estimate the superpixel labels of the rest of the superpixels.
For each label l ∈ L, let x ∈ [0,1]nR×1 denote the corresponding region potentials.
Potential xa corresponds to the probability of superpixel xa to be assigned label l. We
We minimize the following criterion for our superpixel potentials: min. x D(x) = 1 2 P a,bAR(ra,rb)(xa−xb) 2 = 1 2x TLx, subject to xB= 0, xF = 1, (3.12)
whereL=DAR−ARis the unnormalized Laplacian ofAR.
We seek the potential functionxthat minimizes Eq. 3.12. We assume without loss of generality that superpixel are ordered into marked (seeds) and unmarked (non seeds), and
xM, xU correspond to potentials of seeded and unseeded superpixel nodes respectively.
Then,xU that minimizes Eq. 3.12is given by taking the gradient of our cost function and
setting it to zero, which gives:
LUxU =−LTM UxL, (3.13)
as already discussed in the Chapter2. We solve one linear system for each trajectory label
l ∈ Land assign each unmarked superpixel to the label it has the highest potential for.