Once the transformation between the correlated images has been estimated as described in the previous section, one can simply reconstruct an approximate version of the second image ˜I2by warping the reference image ˆI1 using a set of local transformations that forms the warping operator WΛ (see Fig. 5.1). The resulting approximation is however not necessarily consistent with the quantized measurements ˆY2; the measurements corresponding to the projection of the image ˜I2 on the sensing matrix Φ are not necessarily
equal to ˆY2. The consistency error might be significant, because the atoms used to compute the correlation
and the warping operator do not optimally handle the texture information.
We therefore propose to add a consistency term Et in the energy model of OPT-1 to form a new opti- mization problem. The consistency term enforces the image predicted through the warping operatorWΛto
5.5 Consistent image prediction by warping 73
tween the quantized measurements generated from the predicted image ˜I2=WΛ( ˆI1) and the measurements
ˆ
Y2. The consistency term Etis written as
Et(Λ) = ˆY2− Q[Φ˜I2]22= ˆY2− Q[ΦWΛ( ˆI1)]22, (5.15)
whereQ is the quantizer. It should be noted that, when the measurements are not quantized, the consistency term reads
Et(Λ) =Y2− ΦWΛ( ˆI1)22. (5.16)
We then merge the three cost functions Ed, Esand Et with regularization constants α1 and α2 in order to form a new energy model ERfor consistent image prediction, expressed as
ER(Λ) = Ed(Λ) + α1Es(Λ) + α2Et(Λ). (OPT-2)
We now highlight the differences between the terms Ed and Etused in OPT-2. The data cost Edadapts the coefficient vector to consider the intensity variations between images but fails to properly consider the texture information. On the other hand, the consistency term Etwarps the atoms by considering the texture information in the reconstructed image ˆI1 but fails to carefully consider the intensity variations between
images. These two terms therefore impose different constraints on the atom selection that effectively reduce the search space. We experimentally show later that the quality of image ˜I2 is maximized, when all the
three terms are activated in the optimization problem OPT-2. Finally, it should be noted that the data cost in OPT-2 can be replaced with the robust data cost ˜Ed given in Eq. (5.8) to properly account for the measurement quantization noise. It reads as
ER(Λ) = ˜Ed(Λ) + α1Es(Λ) + α2Et(Λ). (OPT-2 (Robust))
In order to solve OPT-2 or equivalently to estimate the correlation model that leads to consistent image prediction, we propose to use the same optimization methods as those described in Section 5.4. In both local and global optimization approaches we modify the objective functions to include the consistency term, since the main difference between the OPT-1 and OPT-2 is the inclusion of the consistent term Et. The steps carried out to solve OPT-2 problem using local optimization methodology is summarized in Algorithm 5 where in line 11 we compute the consistent term Et energy based on warping the reference image (in addition to Ed and Es given in lines 9 and 10). As described in the previous section, such an approach estimates a suboptimal solution with tractable computational complexity.
On the other hand, the global optimization technique based on Graph Cuts estimate a better solution as they usually converge to a strong local minima. In order to solve OPT-2 problem with Graph Cuts we need to compute the cost of assigning a label lk ∈ L to each pixel z. This follows from the fact that atom-wise labeling problem is recast as a pixel-wise labeling problem (see Section 5.4.2). The cost of assigning a label
lk ∈ L to all pixels z in the support of atom gγi (i.e., ∀z ∈ Zi in Eq. (5.12)) is computed using Eq. (5.6) and Eq. (5.16) where Λ = (γ1, γ2, . . . , γi+ lk, . . . , γK). The former one computes the cost of assigning a label lk ∈ L to pixels Zi based on data fidelity and the latter one computes the consistent cost of this label. However, as described earlier the pixels in the overlapped region could be assigned more than one label in the overlapping regions. In such cases, we take the value corresponding the atom index that has maximum response (see Eq. (5.13)). Finally, the data cost Ed in OPT-2 can be further replaced by the robust data cost term ˜Ed given in Eq. (5.8), and it can be solved using local or global optimization techniques described above. We show later that the performance of our scheme can be improved by using the robust data cost term ˜Ed.
Algorithm 5 Local optimization: Correlation estimation with OPT-2
1: Input K, α1, α2, δtx, δty, δθ, δsx, δsy
2: Generate{gγi} from ˆI1 s.t. ˆI1≈!Ki=1cigγi
3: Initialize Λ−1={γi}, k = 0
4: repeat
5: Generate index search space Sk based on Λk−1 (with Eq. (5.14))
6: for all Parameter vectors Λ in Sk do
7: Compute the motion field
8: Warp the reference image ˆI1 using motion field,WΛ( ˆI1)
9: Compute the data term Ed(Λ) with Eq. (5.6)
10: Compute the smoothness term Es(Λ) with Eq. (5.10)
11: Compute the consistency term Et(Λ) with Eq. (5.15)
12: Compute the global energy ER(Λ) = Ed(Λ) + α1Es(Λ) + α2Et(Λ)
13: end for
14: Λk= arg min
Λ∈SkER(Λ)
15: k = k + 1
16: until convergence is reached
Complexity Issues
We discuss now briefly the computational complexity of the correlation estimation algorithm, which can basically be divided into two stages. The first stage finds the most prominent features in the reference image using sparse approximations in a structured dictionary. The second stage estimates the transformation for all the features in the reference image by solving a regularized optimization problem such as OPT-1 or OPT-2. Our framework offers a very simple encoding stage with image acquisition based on random linear projections. The computational burden is shifted to the joint decoder, which can still trade-off the complexity and performance. The decoder is able to handle computationally complex tasks in our framework. However, the complexity of our system can be reduced in both stages compared to the generic implementation proposed above. For example, the complexity of the sparse approximation of the reference image can be reduced significantly using a tree-structured dictionary, without significant loss in the approximation performance [149]. In addition, a block-based dictionary can be used in order to reduce the complexity of the transformation estimation problem with block-based computations. Experiments show however that this comes at a price of a performance penalty in the image quality. Overall, it is clear that the decoding scheme proposed above offers high flexibility with an interesting trade-off between the complexity and performance. For example, one might decide to use the simple data cost Edgiven in Eq. (5.6) even when the measurements are quantized; it leads to a simpler scheme but to reduced estimation accuracy.