Caracterización

So far we have only presented Euler-Lagrange equations, that is partial differen- tial equations (PDEs) in the continuous domain. We need implementations in the discrete domain to actually minimize the energies on digitally sampled im- age data: We need numerical solutions to our problems and have used different well-known methods of discretizing PDEs.

Motion Adaptive Deinterlacing

Our target is to solve the gradient descent equation in (3.7) and we do so explic- itly, using forward difference for the evolution derivative ∂τand central difference

for the divergence terms. (Approximations of continuous derivatives as discrete differences are described in several books, e.g. [89] and [1]). The forward dif- ference of the evolution time derivative in (3.7) is a Taylor approximation of

∂τu = ∂u/∂τ: δ+ τu = ∂u ∂τ = u(x, y, t; τ + ∆τ ) − u(x, y, t; τ ) ∆τ m

u(x, y, t; τ + ∆τ ) = u(x, y, t; τ ) + ∆τ ·∂u

∂τ (3.17)

Using (3.17) all we have to do is to add the time step ∆τ multiplied with the output of (3.7) to the current value of each pixel. For the 2D spatial diver- gence term in (3.7) we have used three different schemes, one using a 4-point neighborhood of the current pixel and two using a full 8-point neighborhood, as described in [1]. These filters are very similar to those described below for the motion compensated deinterlacer. It is a well known (and proven) fact that us- ing gradient descent with spatial filters of the type we use, the time step should be ∆τ ≤ 0.25 in (3.17) and using any values larger in e.g. de-nosing or inpaint- ing will give an unstable solution (typically the whole image will turn uniformly average grey). But in testing our variational MA deinterlacer, we found that even with ∆τ ∈ [0.5 − 1.5] depending on the sequence our scheme stayed sta- ble, owing to the fact that the filter support is mostly fixed pixel values from original, untouched lines.

Optical Flow

We have made our numerical implementation along the lines of [65] and [9] using a fixed point solver to isolate the linear part of the system, which is then solved using a Gauß-Seidel solver run until convergence or until a maximum number of iterations is reached. For the optical flow estimation it is still possible to use a multiresolution setting, which will handle large motions easily and speed up convergence. We thus use multiresolution as in [65] and [9], employing a down- and up-sampling scheme described in [11].

For the fusion-scaling-smoothing, after deriving an Euler-Lagrange equation of (3.16), discretization is done using standard methods. The first term is similar to the E0 in (3.14) just operating on both flow components v1 and v2.

The second term is the same as the E3-term of (3.13) and thus treated in the

same way.

Motion Compensated Deinterlacing

To solve (3.15) we use a Gauß-Seidel solver with a fixed point strategy to lin- earize the nonlinear system as described e.g. in Section 5.3.6 of this thesis. The discretization is as follows. We first look at the spatial term of (3.15) and fol- lowing the ideas of discretization of total variation on the spatial 2D gradient

given in [1], [18], [64] and [88] we can do a simple rewrite div2 µ ∇u ψ(|∇u|2₎ ¶ = divxy(A∇u), A = 1 ψ(|∇u|2₎ (3.18)

remembering that ψ0_{(s)/s = 1/ψ(s). The spatial gradient is}

∇u = (∂u/∂x, ∂u/∂y)T = (∂xu, ∂yu)T. (3.19)

Now from the definition of the 2D divergence we have

divxy(A|∇u|) = ∂x(A∂xu) + ∂y(A∂yu). (3.20)

Using (3.19) and (3.20), equation (3.18) can be approximated finitely as ([1]) divxy(A∇u) ≈ δox,h/2(Aδx,h/2o u) + δoy,h/2(Aδy,h/2o u) (3.21)

where δo

x,h/2 is the central difference approximation of ∂u/∂x and δy,h/2o the

same in the y-direction. h is the distance between two horizontally or vertically neighboring pixel positions (grid points) and most often set to 1. By definition

δx,h/2o = (u(x + h/2, y, t) − u(x − h/2, y, t) h δo y,h/2 = (u(x, y + h/2, t) − u(x, y − h/2, t) h . (3.22)

Before we do the next rewrite we switch to compass coordinates where the indexes w, e, n, s represents the directions in the grid and is short for west, east, north and south, uw= u(x−1, y, t) and so on. ucis the center (or current) pixel.

Plugging (3.22) in (3.21) and rewriting gets us the divergence approximation divxy(A∇u) ≈ 1 h2 ³ (Aw/2· uw+ Ae/2· ue+ An/2· un+ As/2· us) − (Aw/2+ Ae/2+ An/2+ As/2) · uc ´ . (3.23)

We need the A’s at half-grid point, and we get them by calculating A’s at all grid points and then interpolating them at the half-grid points. We could use linear interpolation, i.e.

Aw/2≈ (Ac+ Aw)/2 (3.24) but Aw≈ 2 1 Ac + 1 A0 w (3.25)

endorses edge preservation. Say Ac = 100 (flat region) and A0w≈ 0 (edge), then

the approximation would be Aw= 50 from (3.24) corresponding to a flat region

and Aw≈ 0 from (3.25) corresponding to the close by edge we wish to preserve.

To calculate the A’s at grid points, the choice of ψ(s2_{) =}√_s2_{+ ε}2 _{is clever.}

Since 1/|∇u| = 1/p(∂xu)2+ (∂yu)2 the choice leads to

A ≈ q 1 1 4 ¡ (ue− uw)2+ (un− us)2 ¢ + ε2 (3.26)

when using central differences and is similar to the 2D total variation term used for inpainting by Chan and Shen in [18]. This is also the 4-point neighborhood scheme used for the motion adaptive deinterlacer. The first of its two 8-point scheme also uses the diagonal directions, and the second weighs the 4-point diagonal with the 4-point standard given in (3.26) according to a detection of edge directions. Extensive testing of the performance of the three in [57] showed practically no difference in output quality, and thus we chose to use the 4-point standard scheme in (3.26) only, as it approximately a factor of two faster than the two 8-point schemes.

For the temporal part we have a 1D divergence, div, or simply a derivative along the flow. Since the flow is subpixel precise, the intensities at the position the flow points to is mostly not at grid point and has to be interpolated. We use bilinear interpolation for that. uc is once again the current, center pixel,

ua = u(x + vx, y + vy, t + 1) is the after intensity value along the forward flow

and ub= u(x + vx, y + vy, t − 1) is the before intensity value along the backward

flow and we have

Ba ≈p 1

(ua− uc)2+ ε2

and Bb≈ p 1

(uc− ub)2+ ε2

This lead us to the complete direct solution

uc= λ1(Aw· uw+ Ae· ue+ An· un+ As· us) + λ2(Ba· ua+ Bb· ub)

λ1(Ae+ Aw+ An+ As) + λ2(Ba+ Bb) (3.27)

where the A’s and the B’s are the nonlinear components. In case we include the data term, the full solution becomes

uc=λ1(Aw· uw+ Ae· ue+ An· un+ As· us) + λ2(Ba· ua+ Bb· ub) + χ · u0,c

λ1(Ae+ Aw+ An+ As) + λ2(Ba+ Bb) + χ

(3.28) where u0,cis the current pixel in the input (χ is one for known lines and zero for

new lines). We run a number of outer fixed point iterations where we update the

A’s and the B’s once in each followed by a number of inner relaxation iterations

(the Gauss-Seidel solver) where we update uc by solving (3.27) or (3.28).