Consolidación del

We further evaluate the use of BI modules on the Cityscapes dataset [57]. Cityscapes contains 20K high-resolution (1024_{× 2048) images of street scenes with coarse pixel} annotations and another 5K images with fine annotations, all annotations are from 19 semantic classes. The 5K images are divided into 2975 train, 500 validation and remain- ing test images. Since there are no publicly available pre-trained models for this dataset yet, we trained a DeepLab model. We trained the base DeepLab model with half reso-

7.5 Discussion and Conclusions

Model IoU (Half-res.) IoU (Full-res.) Runtime(s) DeepLab CNN 62.2 65.7 0.3

BI6(2) 62.7 66.5 5.7

BI6(2)-BI7(6) 63.1 66.9 6.1

DeepLab-CRF 63.0 66.6 6.9

Table 7.4: Street scene Segmentation using DeepLab model. IoU scores and runtimes (in sec) of different models on Cityscapes segmentation dataset [57], for both half- resolution and full-resolution images. Runtime computations also include superpixel computation time (5.2s).

(a)Input (b) Superpixels (c)GT (d) Deeplab (e)Using BI

Figure 7.8: Street scene segmentation. Example results of street scene segmentation. (d) depicts the DeepLab results, (e) result obtained by adding bilateral inception (BI) modules (BI6(2)+BI7(6)) between FC layers.

lution images (512_{× 1024) so that the model fits into GPU memory. The result is then} interpolated to full-resolution using bilinear interpolation.

We experimented with two layouts: only a single BI6(2) and one with two inception

BI6(2)-BI7(6) modules. We notice that the SLIC superpixels [6] give higher quantization

error than on VOC and thus used 6000 superpixels using [66] for our experiments. Quan- titative results on the validation set are shown in Tab. 7.4. In contrast to the findings on the previous datasets, we only observe modest improvements with both DenseCRF and our inception modules in comparison to the base model. Similar to the previous exper- iments, the inception modules achieve better performance than DenseCRF while being faster. The majority of the computation time in our approach is due to the extraction of superpixels (5.2s) using a CPU implementation. Some visual results with BI6(2)-BI7(6)

model are shown in Fig. 7.8 with more in Appendix A.5.2.

7.5 Discussion and Conclusions

The DenseCRF [151] with mean field inference has been used in many CNN segmenta- tion approaches. Its main ingredient and reason for the improved performance is the use

Chapter 7 Bilateral Inception Networks

of a bilateral filter applied to the beliefs over labels. We have introduced a CNN approach that uses this key component in a novel way: filtering intermediate representations of higher levels in CNNs while jointly learning the task-specific feature spaces. This prop- agates information between earlier and more detailed intermediate representations of the classes instead of beliefs over labels. Further we show that image adaptive layouts in the higher levels of CNNs can be used to an advantage in the same spirit as CRF graphs have been constructed using superpixels in previous works on semantic segmentation. The computations in the 1_{× 1 convolution layers scales in the number of superpixels which} may be an advantage. Further we have shown that the same representation can be used to interpolate the coarser representations to the full image.

The use of image-adaptive convolutions in between the FC layers retains the appealing effect of producing segmentation masks with sharp edges. This is not a property of the superpixels, using them to represent information in FC layers and their use to interpolate to the full resolution are orthogonal. Different interpolation steps can be used to propa- gate the label information to the entire image, including bilinear interpolation, bilateral upsampling, up-convolutions and DenseCRFs. We plan to investigate the effect of differ- ent sampling strategies to represent information in the higher layers of CNNs and apply similar image-adaptive ideas to videos.

We believe that the Bilateral Inception models are an interesting step that aims to di- rectly include the model structure of CRF factors into the forward architecture of CNNs. The BI modules are easy to implement and are applicable to CNNs that perform struc- tured output prediction.

Chapter 8

Conclusions and Outlook

Generative models provide a strong set of tools for modeling the vision world around us. But their use in computer vision is hampered by the complexity of inference in them causing the vision community to favor data-hungry discriminative models. Both gener- ative and discriminative models have complementary advantages and disadvantages as discussed in Chapter 2. Generative models provide an easy handle for incorporating prior knowledge about the task but inference is often too complex in them. Discrimi- native models, on the other hand, have a straightforward inference scheme as forward evaluation of models, but lack principled ways of incorporating prior knowledge into them.

This thesis work proposed techniques for alleviating some of the key issues with prominent computer vision models by improving inference in them. A common strat- egy that is followed across several techniques proposed in this thesis is leveraging the complementary models for better inference in a given model. That is, we leverage dis- criminative models for better inference in generative computer vision models. And we used generative knowledge (in the form of bilateral filters) and enriched the existing dis- criminative CNN models. This way, this thesis made important steps in bridging the gap between generative and discriminative vision models. The proposed inference tech- niques are flexible enough to deal with different task scenarios (e.g., availability of large or small amounts of data).

Inference in Generative Vision Models In the case of generative models, we leverage discriminative clustering or random forests techniques to accelerate and/or to improve the Bayesian inference. In Chapter 3, we proposed a new sampling technique called ‘Informed Sampler’, where discriminative models help in better exploration of target do- main, via informed proposals, while doing MCMC sampling. In Chapter 4, we proposed a new message passing technique called ‘Consensus Message Passing’ where random forest predictors are used for predicting consensus messages during standard message passing inference resulting in convergence to better solutions.

In both ‘Informed Sampler’ and ‘Consensus Message Passing’ (CMP), we made sure that the theoretical guarantees that come with the well established inference techniques are not violated with our modified inference schemes. In the informed sampler, we

Chapter 8 Conclusions and Outlook

achieve this by injecting discriminative knowledge in MCMC sampling via proposal distributions and adhering to detailed balance condition while sampling. And in consen- sus message passing, we used consensus messages from discriminative predictors only during the first few iterations ensuring that the fixed point reached by our modified infer- ence is also a fixed point of standard message passing in the model. We evaluated both the informed sampler and CMP techniques on three different generative models each, reflecting a wide range of problem scenarios, where we consistently observed improved inference with the proposed techniques in comparison to standard sampling and message passing inference techniques.

Inference in Discriminative Vision Models In this thesis, we focus on the inference in prominent CNN models. Spatial convolutions form the basic building block of most CNN architectures. A key observation in this thesis work is that the bilateral filters [15, 251] are a generalization of spatial convolutions and do not have many of the limitations that spatial convolutions have. The key issue with the existing use of bilateral filters is they are confined to fixed hand-tuned parameterization.

In Chapter 5, we proposed a generalized bilateral filter and devised a gradient based technique for learning the filter parameters. Experiments on wide range of problems showed the superior performance of learnable bilateral filters with respect to using Gaus- sian filter kernel. Learnable bilateral filters enabled us to stack several filters together and learn all of them via back-propagation. Using this, we proposed novel neural network architectures which we call ‘Bilateral Neural Networks’ (BNN).

In Chapter 6, we showed how BNNs can be easily adapted to filter video data for propagating temporal information across video frames. In Chapter 7, we proposed new and fast neural network modules, based on explicit Gaussian bilateral filtering called ‘Bilateral Inceptions’ and showcased how we can modify existing segmentation CNN architectures for big improvements in accuracy while adding little time overhead.

Bilateral filters form the core of mean-field inference in DenseCRF models [151] and provide a way to incorporate prior knowledge about the scene in the form of dense feature-based connectivity across the pixels. By integrating learnable bilateral filters into standard CNN architectures, we brought the worlds of CRF and CNN closer, providing a way to incorporate prior knowledge into CNNs.

8.1 Summary of Contributions

The following list summarizes the specific contributions of this thesis work:

• We devised a novel MCMC sampling approach called ‘Informed Sampler’ (Chap- ter 3) for doing Bayesian inference in complex generative vision models. The In- formed sampler leverages discriminative approaches for improving the efficiency of MCMC sampling. Experiments on a wide range of generative vision models

8.1 Summary of Contributions

showed significantly faster convergence of our sampler while maintaining higher acceptance rates. This opens up possibilities for using complex generative models like graphics engines for addressing vision problems.

• We devised a novel message passing technique called ‘Consensus Message Pass- ing (CMP), in Chapter 4, for doing Bayesian inference in layered graphical mod- els used in vision. Experiments on diverse graphical models in vision showed that CMP resulted in significantly better performance compared to standard message passing techniques such as expectation propagation or variational message pass- ing. Moreover, CMP is the first instance where the Infer.NET [186] probabilistic programming language is shown to be useful for addressing vision problems. • We parameterized bilateral filters as general sparse high dimensional filters (Chap-

ter 5) and devised an approach for learning the filter kernels via standard back- propagation learning techniques. This resulted in a general technique for learning sparse high-dimensional filters, which in turn resulted in a generalization of stan- dard bilateral filters. Experiments on wide range of applications showed improved performance with respect to standard Gaussian bilateral filters.

• We also show how learning bilateral filters can generalize the fully-connected con- ditional random field models (DenseCRF) to arbitrarily learned pairwise potentials (Section 5.5) instead of standard Gaussian pairwise edge potentials. DenseCRF is one of the widely used CRF techniques in vision and this generalization carries forward to most of its existing applications and helps in better integration into end-to-end trained models like convolutional neural networks (CNN).

• Our technique for learning general sparse high dimensional filters also generalizes standard spatial convolutions in CNN frameworks. This opens up possibilities for applying CNNs to sparse high-dimensional data, which is not feasible with many standard CNN techniques. Moreover, our technique can be used for learning image-adaptive filters inside CNNs instead of standard image-agnostic 2D filters. • We adapted the learnable sparse high dimensional filters for video filtering, in

Chapter 6, and proposed a novel neural network approach for propagating content across video frames. We call our networks ‘Video Propagation Networks’ (VPN). Experiments on video object segmentation and semantic video segmentation showed that VPN outperformed existing task-specific methods while being faster.

• In Chapter 7, we devised a new CNN module called ‘Bilateral Inception’ that can be readily inserted into standard segmentation CNN models resulting in better performance while producing the result at original image resolution and also al- leviating some of the need for post-processing. Experiments on state-of-the-art CNN models resulted in significantly better performance with our inception mod- ule while being competitive in time.

Chapter 8 Conclusions and Outlook