In this section, we describe the extension of the learnable bilateral filtering, proposed in Chapter 5 to video data. Several properties of bilateral filtering make it a perfect candidate for information propagation in videos. In particular, our method is inspired by two main ideas that we extend in this work: joint bilateral up-sampling [148] and learnable bilateral filters (Chapter 5). Although, bilateral filtering has been used for filtering video data before [198], its use has been limited to fixed filter weights (say, Gaussian).
Fast Bilateral Up-sampling across Frames The idea of joint bilateral up-sampling [148] is to view up-sampling as a filtering operation. A high resolution guidance image is used to up-sample a low-resolution result. In short, a smaller number of input points yi and
the corresponding features fi are given yi, fi; i= 1, . . . , Nin, for example a segmentation
result yi at a lower resolution. This is then scaled to a larger number of output points
fj; j= 1, . . . , Nout using the bilateral filtering operation, that is to compute the following
bilateral filtering equation:
6.3 Video Propagation Networks y0i= n
∑
j=1 wfi,fjyj (6.2)where the sum runs over all Ninpoints and the output is computed for all Noutpositions.
We will use this idea to propagate content from previous frames to the current frame (all of which have the same dimensions), using the current frame as a guidance image. This is illustrated in Fig. 6.3. We take all previous frame results y1,···,t−1and splat them into
a lattice using the features computed on video frames x1,···,t−1. A filtering (described
below) is applied to every lattice point and the result is then sliced back using the current frame xt. This result need not be the final yt, in fact we compute a filter bank of responses
and continue with further processing as will be discussed.
For videos, we need to extend bilateral filtering to temporal data, and there are two natural choices. First, one can simply attach a frame index t as an additional time di- mension to the input data, yielding a six dimensional feature vector f= (x, y, r, g, b,t)> for every pixel in every frame. The summation in Eq. (6.2) now runs over all previous frames and pixels. Imagine a video where an object moves to reveal some background. Pixels of the object and background will be close spatially(x, y) and temporally (t) but likely be of different color(r, g, b). Therefore they will have no strong influence on each other (being splatted to distant positions in the six-dimensional bilateral space). In sum- mary, one can understand the filter to be adaptive to color changes across frames, only pixels that are static and have similar color have a strong influence on each other (end up nearby in the lattice space). The second possibility is to use optical flow. If the perfect flow is available, the video frames could be warped into a common frame of reference. This would resolve the corresponding problem and make information propagation much easier. We can make use of an optical flow estimate by warping pixel positions(x, y) by their displacement vector(ux, uy) to (x + ux, y + uy).
Another property of permutohedral filtering that we exploit is that the inputs points need not lie on a regular grid since the filtering is done in the high-dimensional lattice. Instead of splatting millions of pixels on to the lattice, we randomly sample or use super- pixels and perform filtering using these sampled points as input to the filter. In practice, we observe that this results in big computational gains with minor drop in performance (more in Sec. 6.4.1).
Learnable Bilateral Filters The property of propagating information forward using a guidance image through filtering solves the problem of pixel association. But a Gaussian filter may be insufficient and further, we would like to increase the capacity by using a filter bank instead of a single fixed filter. We propose to use the technique proposed in previous chapter to learn the filter values in the permutohedral lattice using back- propagation.
The process works as follows. A input video is used to determine the positions in the bilateral space to splat the input points y(i)∈ RD i.e. the features f (e.g. (x, y, r, g, b,t))
Chapter 6 Video Propagation Networks
that lie on the permutohedral lattice, with dimensionality ysplatted ∈ RD. In effect, the
splatting operation groups points that are close together, that is, they have similar fi, fj.
All lattice points are now filtered using a filter bank B∈ RF×Dwhich results in F dimen- sional vectors on the lattice points. These are sliced back to the Nout points of interest
(present video frame). The values of B are learned by back-propagation. General pa- rameterization of B from previous chapter allows to have any neighborhood size for the filters. Since constructing the neighborhood structure in high-dimensions is time con- suming, we choose to use 1× 1 filters for speed reasons. This makes up one Bilateral Convolution Layer (BCL)which we will stack and concatenate to form a Bilateral Net- work. See Fig. 6.3 for an illustration of a BCL.
BNN Architecture The Bilateral Network (BNN) is illustrated in the green box of Fig. 6.2. The input is a video sequence x and the corresponding predictions y up to frame t. Those are filtered using two BCLs with 32 filters each. For both BCLs, we use the same features f but scale them with different diagonal matrices fa= Λaf, fb= Λbf.
The feature scales are found by cross-validation. The two 32 dimensional outputs are concatenated, passed through a ReLU non-linearity and passed to a second layer of two separate BCL filters that uses same feature spaces fa, fb. The output of the second filter
bank is then reduced using a 1× 1 spatial filter (C-1) to map to the original dimension of y. We investigated scaling frame inputs with an exponential time decay and found that, when processing frame t, a re-weighting with(αyt−1, α2yt−2, α3yt−3···) with 0 ≤ α ≤ 1
improved the performance a little bit.
In the experiments, we also included a simple BNN variant, where no filters are applied inside the permutohedral space, just splatting and slicing with the two layers BCLa and
BCLband adding the results. We will refer to this model as BNN-Identity, it corresponds to an image adaptive smoothing of the inputs y. We found this filtering to have a positive effect and include it as a baseline in our experiments.