DEFINIR LINEAMIENTOS PARA EMPRESA JUNIOR UCSM – PROCESO

The image formation process was described in Chapter 2. Here an original model for the scene reconstruction problem was presented, describing the relationship between camera data and a discrete model of the scene. This highlighted some of the problems with the binary opacity assumption, showing this can cause problems around oblique surfaces or at depth discontinuities. It was also shown that the formation of pixel intensities could be approximated by an integral or discrete summation of the band limited scene opacities and radiances along a line in space, simplifying the more exact 3D integral.

Additionally, the concept of the imaging convolution kernel was considered. It was shown that differences in the convolution kernels lead to variations in observed intensities between images. An alternative multiple camera approach for dealing with these variations was later presented in Chapter 6.

An overview of reconstruction techniques was provided in Chapter 3. The use of a statistical approach to deal with the scene reconstruction problem was introduced and the differences between MAP and MMSE were highlighted. It was discussed how traditional stereo matching could be performed using a volumetric scene model.

Chapter 3 also demonstrated that region based matching and many of the techniques for dealing with non-Lambertian surfaces were equivalent to filtering individual voxel likelihoods within a volumetric model. Techniques for dealing with visibility interactions were introduced, and a variety of optimisation techniques which could be applied to the scene reconstruction problem were discussed.

The problems posed by occlusions were dealt with in Chapter 4. It was demon- strated that the joint probability distribution could be expressed as a product of inde- pendent terms, if the visibility of the opaque voxels was known. This allowed the MAP estimate to be expressed as a summation of independent data error terms correspond- ing with the negative conditional log probabilities. The concept of a complete scene estimate was introduced. This was defined as an estimate where at least one opaque voxel or surface is required along each pixel ray in every image. The summation of inde- pendent data error terms could then be formulated as a pixel ray assignment problem, where the objective was to assign at least one opaque voxel along every pixel ray so that

the sum of error terms was a minimum.

In Chapter 4 an iterative approach for dealing with visibility updating was also introduced. Here it was shown that a greedy selection process was more appropriate with visibility updating than the described pixel ray assignment algorithm.

In the work of Preddey and Lane [1997] and Harding et al. [2000] the assignment of voxels was based solely on the projected square error of each voxel. In this work, some simple and improved techniques for reliably assigning opaque voxels were developed. Prior information was used to assist opaque voxel assignment. The results in this chapter highlighted some of the problems with the binary opacity model, motivating the development of a pixel dissimilarity measure presented in Chapter 6. It was also found that a combination of both smoothing and visibility priors produced the best results. A hierarchical approach for efficiently calculating the most likely voxel at each iteration was presented.

To improve the use of prior information and obtain a global optimum, belief propa- gation was applied. An improved volumetric model of the scene was presented to model the visibility interaction between scene variables and incorporate prior information. This was represented using a factor graph model describing the joint probability of the imaging system. It was shown that belief propagation could be applied to this model to find the MAP estimate of the scene. The local structure of the probability distribution within the model was utilised to compute the message updating more efficiently for this particular volumetric model. However, the resulting algorithm was found to be unstable and a simple technique for helping convergence was developed.

The results in Chapter 5 were promising, but the model was very memory intensive and computing messages at each iteration time consuming, in part due to the volumetric nature of the model. Nevertheless, one reason for investigating belief propagation over other optimisation techniques is that it is highly parallelisable, lending itself to efficient implementation on parallel architectures.

To avoid some of the problems encountered with the volumetric model presented in Chapter 5, a dynamic approach to modelling the scene was presented, where the local probability distributions were iteratively updated to reflect the visibility between scene variables. To ensure the scene estimate was complete a new known-visibility volumetric model was presented. However, it was found to be unstable using the max product belief propagation algorithm to optimise the model. The dynamic updating was also applied to an alternative simpler single depth map model, with promising results, showing that this approach can be used to improve the scene reconstruction process.

7.1 RECOMMENDATIONS FOR FUTURE RESEARCH

Future work will focus on improving the system model, as well as attempting to increase the speed of the existing algorithm.

7.1 RECOMMENDATIONS FOR FUTURE RESEARCH 147

7.1.1 Improving system modelling

Work could be done to improve modelling of the imaging system, or techniques for identifying the errors that occur during modelling could be developed. The most signif- icant problems are sampling variations caused by differences in the imaging convolution kernel, as well as problems with the binary opacity assumption in the vicinity of steeply sloping surfaces or depth discontinuities. Specular reflections are another common cause of reconstruction error.

Variations between the modelled and observed intensity of points degrade the re- construction if not properly accounted. Opportunity for future research is significant. Some ideas include identifying depth discontinuities and adjusting the modelled joint probability distribution accordingly. The modelled noise level could then be reduced as the system model is improved. An improved pixel dissimilarity measure for multiple cameras also needs to be developed, possibly taking into account the local slope in the observed intensity functions.

7.1.2 Developing application of prior information

Improved results can also be obtained through more detailed application of prior infor- mation. This has been observed in the performance of recent reconstruction algorithms. For example, on the Middlebury test set1 all the currently top-performing algorithms use some form of image segmentation based on the relationship between observed inten- sity and scene structure. As applied to the volumetric models presented in this thesis, surface priors may prove more effective than volumetric priors.

7.1.3 Improving global optimisation techniques

Another place for significant improvement is in the use of more effective and efficient global optimisation techniques for optimising the joint probability distribution. Recent techniques such as tree re-weighted message passing and graph cuts could prove more successful depending on the structure of the joint probability distribution. Continuous optimisation techniques such as expectation propagation could also be considered. Per- haps the best approach will be to use one technique to form a rough global optimum and then refine this using a stronger but more local optimisation technique.

7.1.4 Efficient implementation

Improvements could also be made to the efficiency of optimisation algorithms through better implementation and custom hardware. Implementation of belief propagation on custom hardware would allow the computation of messages to be performed in parallel.

It should be noted that the belief propagation can be performed in the logarithmic do- main thereby replacing multiplication with addition. This, in conjunction with parallel implementation, makes it suitable for fast hardware implementation.

For sequential implementation on a standard PC, the efficiency of the belief prop- agation algorithm could be improved by appropriate synchronisation of the message updating, or performing more updates in regions where beliefs are changing rapidly.