La capacidad de adaptación Su concepción y alcance profesional

3.2

Reconstructability of NRSfM

To characterize the possibility in recovering the non-rigid shape and camera motion given an input video, we propose to analyze the camera motion and 3D shapes. In the following paragraphs, we first review the reconstructability proposed for trajectory reconstruction. Then we extend the concept to general NRSfM and propose our reconstructability evaluation based on 3D shape similarity.

Reconstructability in trajectory reconstruction: Given available camera motions, trajectory reconstruction aims at estimating a 3D point trajectory from a 2D feature track. Park et al. [21] proposed a measure on the possibility of reconstruction, namely “reconsructability”, by analyzing the correlation between camera trajectory and moving point trajectory. They used a perspective camera model, which is defined as:

" xi 1 # 'Pi " Xi 1 # , or " xi 1 # × Pi " Xi 1 # = 0 (3.1)

where Xi is a point’s 3D coordinate, xi is its 2D projection, Pi is a 3×4 cam-

era projection matrix, and [·]× is the skew symmetric representation of the cross

product. Since the above equation is defined up to scale, x can be replaced as follows: " Pi " Xi 1 # # × Pi " ˆ Xi 1 # = 0 (3.2)

Assuming the relative camera locations are estimated beforehand, the camera ma- trix can be normalized to Pi = [I3| −Ci], where Ci is the camera center. Substi-

tuting Pi to the equation results in:

ˆ

Xi =aiXi+ (1−ai)Ci (3.3)

where ai is an arbitrary scalar. Provided a pre-defined trajectory basis Θ, the

equation can be solved in a least squares manner: min

ˆ

β,A

kΘβˆ−AX−(I−A)Ck. (3.4)

where β represents trajectory basis coefficients. The authors then decompose the point trajectory and camera trajectory into the column space and null space of Θ: X = ΘβX +Θ⊥βX⊥, C = ΘβC +Θ⊥βC⊥. Based on this decomposition, recon-

3.2 Reconstructability of NRSfM 24

structability is defined as:

η= kΘ

⊥

β_C⊥k

kΘ⊥β_X⊥k (3.5)

It is then proved that as the reconstructability approaches infinity, the recon- struction accuracy is increased. From the expression ofη, it is clear that increasing kΘ⊥β_C⊥k and decreasing kΘ⊥β_X⊥k is going to enhance the reconstructability. This shows that in general, when point trajectories are smooth and camera trajectories are fast and random, accurate reconstruction is likely to achieve.

In practice, reconstructability enhancement can be done by choosing a DCT basis that makes camera trajectory lie in its null spaceΘ⊥, while keeping the ability to express the point trajectory. Although this perfect condition is not likely to be reached, one can create a specialized DCT basis, which is a projection of the original DCT onto the null space of the camera trajectory. However, this method requires a known camera trajectory, which is not given in a realistic NRSfM problem.

Reconstructability for NRSfM: To extend the concept of “reconsructabil- ity” from trajectory reconstruction to general NRSfM, we need to measure the complexity in both camera motion and 3D shape variation.

Shape complexity: Given a primitive non-rigid shape S, its complexity (re- constructability) can be well characterized by the rank, i.e., ηS =rank(S]), which

is determined by the number of principal components in PCA analysis that rep- resent 90% of the energy. S] is the re-ordered version of S defined as in equation 2.21.

Motion complexity: Under our formulation, camera motion only consists of the rotation component. As camera rotation resides in a manifold as Ri ∈SO(3),

to define the complexity of camera motion, we need to characterize the distance on the manifold. To ease the computation, we use the chordal distance to evaluate the difference between rotations as: dij =kRi−RjkF. In this way the global motion

complexity could be defined as: ηR=P_i,jd2ij.

By putting the shape complexity and the motion complexity together, we obtain the “reconsructability” for general NRSfM as :

η(R,S) = ηR(R)

ηS(S)

. (3.6)

According to the definition, a larger motion complexity and a smaller shape com- plexity will generally result in a higher reconsructability, which is consistent with existing work in NRSfM [27] [33].

3.2 Reconstructability of NRSfM 25

(a) Constant camera speeds in random direc- tion

(b) Completely random rotation

Figure 3.2: Numerical experiments analyzing the relationship between shape complexity, motion complexity and 3D reconstruction performance. (a) 3D reconstruction error on the “Triangle” sequence with varying shape complexity under different camera rotation speeds. (b) 3D reconstruction error for different UMPM sequences with varying shape complexity under completely random camera motions.

Numerical examples: To evaluate the validity of our reconstructability for NRSfM, we set up a series of experiments on the UMPM sequences [1] to analyze the relationship between reconstructability and motion/shape complexity. The 3D results are performed using PND [2].

To obtain sequences with varying shape complexity, we project ground truth UMPM 3D shapes into low dimensional subspace by varying dimensionK. Then we perform a Procrustean alignment to the sequence such that all frames are aligned to the first frame, thus eliminating the rigid component in non-rigid shape de- formation. To accurately test the theoretical correctness, we apply two different kinds of camera motions in our experiments: 1). varying rotation speed (from 0.1 degree to 3 degrees per frame, thus varying camera motion complexity) with a random direction following a Gaussian distribution at each frame; 2). completely random camera rotations at each frame, for which the camera motion complex- ity has been maximized. Experimental results are illustrated in Fig. 3.2, where the two figures correspond to the camera motion configurations. In the case of varying camera rotation speed as shown in Figure 3.2(a), 3D reconstruction error generally increases with the increase of shape complexity (rank) and decreases with the increase of rotation speed. In the case of completely random camera motions as shown in Figure 3.2(b), as shape complexity increases, 3D reconstruction error increases correspondingly.

Our previous experiments have shown the impact of random direction rotations. However, in reality, many videos are taken with cameras parallel to the ground,

3.2 Reconstructability of NRSfM 26

(a) Constant camera speed around y-axis (b) Varying camera speed around y-axis

Figure 3.3: Experiments with more realistic camera motions. (a) 3D reconstruction error on the “Table” sequence with varying shape complexity under constant camera rotation speeds. (b) 3D reconstruction error on the “Table” sequence with varying shape complexity under random camera rotation speeds.

i.e. around the y-axis. Therefore, we perform two additional experiments based on this fact: (a) the camera rotates around the axis of human body (y-axis) on a constant speed ofr per frame, (b) the camera rotates around y-axis on a uniformly distributed speed between [0,2r]. That is, for each frame the camera rotates a random angle within [0,2r]. Figure 3.3(a) shows the constant speed case on “Ta- ble” sequence, which is the most complex in our dataset, with speed varying from 0 to 10, and rank from 2 to 9. It shows a trend that error decreases when rota- tion speed increases, but stops decreasing when rot > 1. This trend shows that reconstructability increases when camera rotation increases, but comes to satura- tion when rotation is large enough. Another trend is that error increases when rank increases, except the case where rotation is so small that non-rigid motion provides more information about the 3D object. Note that there is an outlier at

k = 8, rot = 10 because of some special features of the sequence. Figure 3.3(b) shows the random speed case on “chair”, and the legends show the average camera speed. Generally error increases when rank increases and rotation speed increases, but outliers exist because of the features of the specific features of each individual sequence and a small rotation speed. All these experiments demonstrate that our new reconstructability clearly captures the essence in achieving better 3D recon- struction through evaluating shape complexity and motion complexity.