Instintos del yo e instintos sexuales.

The main contribution of this thesis has been to investigate the usability of variational methods in an general applicable upscaling video processor. Bayesian inference has been used to impose regularity of the ill-posed problems of image sequence restoration and enhancement. We have not tested a wide range of different probability functions, but focussed on the use of total variation, which composes a nice balance between on one side mathematical tractability and computational cost and on the other side fidelity in modelling and produced output quality. Since our Bayesian framework and the derivation of variational methods from it was used earlier for inpainting by Lauze and Nielsen in [65], and since variational optical flow methods are well-known in literature, it has mainly been the application and adaption of methods to the upscaling problems that has been new. We will now summarize the contributions in detail chapter by chapter.

Chapter 2: Background

We have discussed how properties of the human visual system dictate the need of high quality upscaling to improve the viewing experience (if no high definition source material is available). High quality image sequences enables the viewer to focus on the journalistic, artistic and entertainment aspects of watching motion pictures, video and television to a degree where technology becomes invisible.

The use of state of the art variational optical flow methods in motion com- pensated upscaling is introduced, and the optical flow methods are integrated in our Bayesian framework for simultaneous optical flow and intensity calculations, from which a variational energy minimization formulation is derived.

Chapter 3: Deinterlacing

We show that no motion adaptive deinterlacer including our variational one can solve The Interlacing Problem of highly detailed regions in motion. The Inter- lacing Problem can only be solved by motion compensated methods gathering information temporally along the optical flow field.

We analyze and discuss the problem of computing optical flows on interlaced video for motion compensated deinterlacing. We chose a strategy computing the flow from the interlaced data alone and use that as input for variational motion compensated deinterlacing.

The Interlacing Problem is in all test cases solved by our variational mo- tion compensated deinterlacer, which yields high quality results with hardly any artifacts present and close to the ground truth. Still, there is room for im- provements in quality, the biggest improvement most likely to come from doing simultaneous calculation of flow and intensities.

Chapter 4: Video Super Resolution

The variational motion compensated video super resolution method presented in Chapter 4 does do simultaneous flow and intensity calculations, producing better results than our earlier non-simultaneous variational video super reso- lution method presented in [54]. Since the latter non-simultaneous method do not compute precise high resolution flow, it does not benefit from temporal information to the same degree as the simultaneous method.

As data term in our model we use the super resolution constraint, which is derived from the image acquisition model, and controls the diffusion process by projecting the suggested energy updates back onto the true solution hyperplane. Our simultaneous video super resolution method is in terms of output quality clearly better than bicubic and bilinear interpolation (the latter widely used in video processing devices today) and is also shown to outperform the super resolution methods of highly expensive film post production and editing systems (but only on one test example so far).

There are super resolution methods in literature that are likely to perform better than our simultaneous variational video super resolution, but due to limitations in what types of depicted objects (e.g. faces only) or flows (e.g. parametric flow only) the applied models allow – or the need for multiple camera recordings of the scenes – these methods are not applicable to general video with arbitrary (natural) content and motion as our method is.

Chapter 5: Temporal Super Resolution

The presented variational motion compensated temporal super resolution meth- od derived from our Bayesian framework simultaneously computes flows and intensities in a multiresolution setting. No other TSR method has computed flow and intensities of nonexisting frames simultaneously, but just estimated flow fields in new frames as a preprocessing step to the intensity calculations.

In Chapter 5 we went into more details about the human visual system than in the background chapter (Chapter 2) to find out what requirements are put on frame rates in modern cinemas and video display systems.

Derived from our variational temporal super resolution framework two ver- sions of variational frame rate doubling are implemented and tested. In the first

version we used the gradient constancy assumption in the flow energy minimiza- tion as done by Brox et al. in [9], but to get a theoretically consistent and less complex algorithm, we leave out the GCA in the other version. The latter is expected to give smoother and less precise flows.

We do not always create perfect new frames, still both versions of variational motion compensated temporal super resolution do produce high quality 50 fps video from 25 fps video without noticeable artifacts during video playback and thus reestablish the pi-effect and the illusion of motion pictures for the problem case of high contrast edges in motion. The version using GCA in the flow energy minimization does give a sharper output in case of object motion, but the difference is not perceived when playing back the frame doubled results as video.

Although we have only implemented and tested a frame rate doubler, imple- mentation of a generic frame rate converter from our variational formulation of the temporal super resolution problem is straightforward.

To truly prove the quality of our results, both for TSR and our two other upscaling methods, we need to compare our outputs with those obtained using other motion compensated upscaling methods. The problems of doing bench- marks is discussed in Chapter 5.

We also discuss the general problem of handling certain types of complex flows with our current implementation of variational optical flow. Suggestions on how to solve the problems are given.

Chapter 6: Detecting Interlaced or Progressive Source of Video

We have designed, implemented and tested of a method for detecting input scan formats. This is a crucial preprocessing step to any video upscaling system as it decides on whether or not to deinterlace the input video signal. Wrong interlaced/progressive classification of the input could have severe consequences to the final output quality of the upscaling system. Our method was shown to classify correctly in more than 98% of all cases, only failing in cases where the image content and motion were of a character that would not lead to bad output quality in case of wrong processing.