Evaluación de los niveles de iluminación

We build a Markov Random Field model [15] on the boundary fragments and superpixels w.r.t. the graph in Section 4.2.1, which defines a joint distribution over the glass labeling given an RGBD image input. Note that our output includes both boundary and region labeling – with which we are able to encode the spatial dependency in a more expressive way. We first introduce the energy function of our model and then describe its components in detail.

Let the boundary fragments be E = {eij} and its subgraph be (VE, GE). Similarly we

have D = {di} and (VD, GD) for superpixels. We define the state space of di as Di= {0, 1},

indicating glass and non-glass. For boundary variable eij, we first assign a direction to it and

define its left and right side. eij is 0 if it is not a glass-vs-nonglass boundary, +1 if the glass

region lies at left side and −1 otherwise. Therefore the state space for eijis Eij= {0, +1, −1}.

The energy function we propose can be written as follows: E = X ij∈VE φE(eij; I ) | {z } boundary unary +β X (ij,kl)∈GE ψE(eij, ekl; I ) | {z } boundary pairwise + γ X i∈VD φD(di; I ) | {z } superpixel unary +λ X (i,j)∈GD ψD(di, dj, eij; I ) | {z } superpixel pairwise (4.2)

where I is the input image, and β, γ and λ are weighting coefficients. The factor graph is shown in Figure 4.4.

from a classifier based on local cues:

φE(eij; I ) = − log(P (eij|fij)) (4.3)

where fij∈ RN is the local feature vector for the boundary fragment eij. We evaluate two dif-

ferent local classifiers: a Support Vector Machine (SVM) with a Radial Basis Function (RBF) kernel and a Random Forest (RF) classifier. The classifier input consists of features extracted from both sides of a boundary fragment. In particular, we extract features from multiple pairs of feature windows, each on either side of the boundary fragment. See Section 4.3.1 for details. The features used for training the boundary unary classifier include:

• Hue and Saturation [135]: This feature is designed to measure the color similarity be- tween both sides of a boundary fragment, as color on both sides of a glass boundary tends to be similar. A twenty-bin histogram is constructed for hue and saturation val- ues on both feature windows. The histograms are then normalized and the Euclidean distance between them are used as our feature.

• Blurring [135]: This feature quantifies the relative smoothness between both sides of a boundary fragment, as glass surface could have a blurring effect on the background. The discrete cosine transform is used and the mean of frequency coefficients is chosen as an indication of smoothness. After this, we use the difference of mean frequencies on both sides as our feature, which reflects the relative smoothness. As this measure can be less reliable on highly textured regions, the measured frequency difference is normalized by a texture entropy measure (i.e., the standard deviation of intensity values on the smoother side of the boundary).

• Blending and Emission: The feature is based on the overlay assumption of glass sur- faces [3] and particularly, the linear model for the intensity of a transparent surface:

I = αIB+ e (4.4)

where IB is the intensity of the background, α is a blending factor, and e is the emis-

sion of the semi-transparent surface. We follow the method in [135] by clustering the intensities on both sides of the boundary and solving for α and e as a linear least square problem.

• Texture distortion [135]: The feature measures the similarity of texture between both sides of a boundary fragment. In particular, texture can be magnified or skewed when

observed through glass. We use a filter bank as described in [130] to obtain a distribu- tion of filter outputs. The texture similarity is then measured by the Euclidean distance between the distributions observed on both sides of the boundary.

• Missing depth: This feature exploits the fact that depth readings of an RGBD sensor tend to be missing on the glass side of the glass boundary while being valid on the other side. Therefore, we can firstly compute the depth missing ratio of a feature window:

missing ratio =N o. of pixels with missing depth reading

N o. of pixels in the f eature window (4.5) Once the missing ratios of a pair of feature windows are obtained (one on each side of a boundary fragment), we use their difference as the feature value. The underlying assumption is that, for non-glass regions the missing ratios from both sides should be low, while for glass regions they should be both high. Generally, a large difference in missing ratio may only be observed near glass/non-glass boundaries.

For boundary fragment orientation we train a separate SVM classifier. We use only two features: saturation and depth missing ratio. The two features were found to be quite robust in identifying boundary orientation. We assign an associated direction to each boundary fragment (i.e., viewed as a vector on the 2D image plane) so we can unambiguously define its left and right. We compute features on the left and right patches respectively, and then subtract the right from the left.

The boundary unary potential is illustrated in Figure 4.1 (d). Each fragment is assigned with a probability for glass object contour (i.e., the darker the more possible), and the orienta- tion is marked with red arrows pointing towards detected glass regions.

Boundary pairwise potentials. The boundary pairwise potential imposes a direction-sensitive smoothness prior. Note that for each boundary fragment eijthere are three possible states. The

model prefers configurations where connected boundary fragments have the glass region on the same side. More formally, we define the smoothness prior for two connected boundary fragments eij and eklas:

ψE(eij, ekl) = 1 − δ(eij= ekl6= 0)

+ C1δ(eij= ekl= 0) + C2δ(eij6= ekl) (4.6)

where δ(·) is the indicator function, and we choose C1 = 0.3 ∗ δ(π₂ < α ≤ π) , and C2 =

Figure 4.5: Illustration of our angle preference for boundary pairwise term. In the top and middle examples, the angles between connected boundary fragments are obtuse and straight respectively. These are commonly found in ground-truth glass boundaries. In the bottom example, however, the angle is acute and is more likely a result from incorrectly identified

glass boundary.

We prefer configurations where the angle between two neighboring boundary fragments are obtuse, so additional penalty terms are added if there is no glass boundary (i.e., eij= ekl= 0)

or the boundary orientation is incompatible (i.e., eij 6= ekl). If the angle is acute, we simply

treat all states equally except if the orientation is compatible (i.e., eij= ekl6= 0). See Figure 4.5

for an illustration of our angle preference.

Superpixel unary potentials. This term is similar to the boundary unary term except that fea- tures are extracted from triangular superpixels. Similar to the boundary orientation classifier, only saturation and depth missing ratio are used. This is because other features we exper- imented with are less effective, particularly when compared to the depth missing cues. As shown in Figure 4.4, we denote the local feature vector for superpixel diwith gi. The result is

illustrated in Figure 4.1 (e).

Superpixel pairwise potentials. This pairwise term specifies valid configurations of a bound- ary fragment and its neighboring superpixels. Any incompatible state will be penalized. Specif- ically, for boundary fragment eij let di be the superpixel that resides to its left and dj to the

right. We set the pairwise potential as:

ψD(di, dj, eij) = δ(di6= dj, eij = 0)

− δ(di= 0, dj 6= 0, eij = +1)

Figure 4.6: Illustration of our superpixel pairwise term. Assume the arrow points towards glass regions in red, and the non-glass regions are in blue. See text for details.

The three terms correspond to the bottom, top and middle examples in the illustration in Figure 4.6. The top and middle examples in Figure 4.6 are two scenarios where the orientation of the boundary fragment (middle column) and the neighboring superpixels (right column) are consistent, so we encourage them by adding negative energy terms. In the bottom example, the labels of two neighboring superpixels are different, but the boundary fragment in between is not part of the glass boundaries (i.e., eij = 0, illustrated as without an orientation arrow). This

is in violation with our graph consistency assumption so a penalty (positive) term is added.