Diffraction-limited optical systems coupled with sensors of various geometries lead to a variety of possible point-spread function shapes [11, 15, 50]. Unfortunately, except for rare super-resolution applications where the imaging hardware can be entirely calibrated ahead of time and is then used to image a static scene, most super-resolution applications face some uncertainty about the exact shape of the point-spread function. In a few cases, the blur can be estimated by measuring the blur on step edges or point light sources in the image, or by taking additional calibration shots [76], but for most applications, determination of the point-spread function is a hard problem.
Blind image deconvolution is a closely-related problem, in which the goal is to take an image blurred according to some unknown kernel, and separate out the underlying source image and the kernel image, using no further inputs. A review of Blind Image Deconvolution is given in [68], though many of the methods are not entirely suitable for real super-resolution applications; for example, zero sheet separation is highly sensitive to noise and prone to numerical inaccuracy for large datasets, and the performance of frequency domain methods also suffers when im- age noise masks the nulls in the image Fourier transform which would otherwise contain information about the blur function. The two most appropriate methods for super-resolution PSF determination are those with the best ability to cope with image noise, and these are Reeves and Mercereau’s “Blur identification by general- ized cross-validation” [99], and Lagendijk et al.’s “Maximum Likelihood image and blur identification” [69].
Cross-validation approaches are widely used to tune model parameters by examining how well a model learned on one subset of the data performs on another disjoint subset, and Generalized Cross-Validation (GCV) extends the leave-one-out variant of this (where the data used in evaluation consist of each element alone in turn). The GCV blur identification method was taken up in multi-frame image super-resolution by Nguyen et al. [79, 80, 81, 82, 83], who consider the blur identification and im- age restoration problem as a special case of image super-resolution where the low- resolution pixels all lie on a fixed grid. They also extend the blur-learning method to handle the learning of a prior strength parameter for image super-resolution.
The maximum likelihood approach for blur identification is used both in Tipping and Bishop’s “Bayesian image super-resolution” [112], and in Abadet al.’s “Param- eter estimation in super-resolution image reconstruction” [1] (and similarly [75]). Both pieces of work begin with a Gaussian data error function and a Gaussian im- age prior over the super-resolution image, then integrate the super-resolution image out of the problem to leave an equation that can by maximized with respect to the variables of interest. In the case of [112], this is the PSF standard deviation and some geometric image registrations; for [1] this is a selection of PSF values and a parameter governing the strength of the prior on the high-resolution image.
Wanget al.[120] use a MAP expression for the PSF parameter, a strong patch- based high-resolution image prior, and importance re-sampling to obtain values for the PSF parameter from its approximate distribution.
Recently, Variational Bayes has been applied both in the field of blind image deconvolution and in super-resolution. Original work by Miskin and MacKay [73] deals with blind source separation and blur kernel estimate. This is extended by Fergus et al. [39] to cover a much wider class of photographic images. The de-
Figure 2.8: Molina et al.’s blind deconvolution using a variational approach to recover the blur and the image. Left: astronomical image displaying signifi- cant blur. Centre and Right: Two possible reconstructions; images taken from [74]. convolution approach of Molina et al. [74] is also similar to Miskin and MacKay’s deblurring work, and builds on the Variational Bayes approach to blur determina- tion in super-resolution. Figure 2.8 shows a blurred astronomical image, and two possible restored images from [74].
2.6.1
Extensions of the simple blur model
Most of the PSF determination techniques mentioned above relate to the estimation of parameters for blurs from specific families like Gaussians or circular blur functions, and only the variational work [39, 73, 74] builds up a non-parametric pixel-wise representation of the blur.
In between the simple one-parameter blur model and the full pixel-wise represen- tation, several bodies of work model motion blur in addition to Gaussian or circular PSFs, making use of the motion of an input video sequence [7, 24, 50, 98].
The image blur plays a key role in depth-from-defocus, where several differently- defocused images are used to estimate a depth map. The extent of the spatially varying PSF for any point in the image is related to the depth of the object in the scene, and some camera parameters. Given that the depth stays the same while the camera parameters are varied, one can acquire several samples with different parameters and then eliminate the unknowns. The problem can be solved in several
ways, both in the image domain with MRFs and in the spatial frequency domain [95, 96]. The main drawback of this approach is that the available data must include several differently-focused frames, and suitably large changes of focus may not occur in a general low-resolution image sequence.