CRITERIOS DE DISEÑO ESTRUCTURAL

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Recall Precision

Recall−Precision −− Man made object categories car motorbike bicycle bus 0 0.2 0.4 0.6 0.8 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Recall Precision Natural categories person cow sheep (a) (b)

Fig. 5.7.Overall recall-precision curves for the different object classes in the PASCAL 2006 VOC challenge. (a) Results on man made objects. (b) Results on natural objects.

these classes. For example cat appear in a wide variety of body poses and orientations. On the other hand, our detector gave the best results in the object detection competition for the person, car, motorbike and sheep classes. For all of these classes the detector outperformed the other competing approaches by a significant margin. It was also ranked second for the bicycle class, being outperformed only by an AdaBoost based classifier that also used HOG descriptors. The performance remained comparable to the best detectors for buses and cows. A detailed com- parison of our detectors with other approaches on the PASCAL test set is given in Everingham et al. [2006a]. Unlike other participants who learned different classifiers for different views such as left, right, front and back views, we learned only one linear classifier for each object class irrespective of the variation in views. This clearly highlights the discriminative power of the HOG descriptors.

Figure 5.8 and Fig. 5.9 show, respectively, the features that R-HOG descriptors cue their decisions on for motorbikes and cars. For motorbikes the blocks corresponding to the motor bikes wheel and rim are the most important, while for the cars roof, wheels and front/rear bonnets are the most important.

5.6 Conclusions

This chapter detailed our object detection and localisation algorithm for classification window scanning detectors. Although such object detectors abound in the literature [Rowley et al. 1998, Papageorgiou and Poggio 2000, Mohan et al. 2001, Fleuret and Geman 2001, Schneiderman and Kanade 2004, Viola and Jones 2004], we present a generalised method for the fusion of overlapping detections. The algorithm not only fuses multiple detections at nearby locations and scales and incorporates detection confidences, but also allows the detection of overlapping object instances occurring at very different scales.

We studied the influence of all of the parameters that affect the performance of the fusion algorithm. The most important conclusion is that as the parameters change, the performance varies substantially even when exactly the same binary classifier is used for all evaluations. In

68 5 Multi-Scale Object Localisation

(a) (b) (c)

(d) (e) (f)

Fig. 5.8.The feature information encoded by R-HOG for motorbikes. (a) The average gradient image over left-sided views in normalised motorbike images. (b) Each “pixel” shows the sum of all of the positive SVM weights in the block centred on the pixel. (c) Likewise for the nega- tive SVM weights. (d) A test image. (e,f) The R-HOG descriptor weighted respectively by the positive and the negative SVM weights. Only the dominant orientation is shown for each cell.

(a) (b) (c)

Fig. 5.9.The feature information that the R-HOG detector cues on for cars. (a) The average gradient image over 104×56 normalised images from the Pascal 2006 VOC training set. The data set contains cars with front/rear, side and in-between views. (b) Each “pixel” shows the sum of all of the positive SVM weights in the block centred on the pixel. (c) Likewise for the negative SVM weights. For positive blocks, the car roof, wheels and front & rear bonnets weigh the most, whereas for negative ones the remaining regions weigh more.

Chapter 4 we concluded that fine grained features help to improve performance, here we find that a multi-scale approach based on fine position and scale sampling gives best results. Ideally the test image should be scanned at all possible scale levels and all positions for each scale level, but this is impractical. A scale ratio of 1.05 and a window stride equal to the cell size or block stride seems to provide a good compromise. Values more than these significantly decrease the performance.

Another conclusion is that the amount of spatial smoothing when fusing detections in 3-D position and scale space should be conservative. Large amount of smoothing always decrease the performance. A value equal to the window stride provides good results. Interestingly, the spatial smoothing can be anisotropic; the best results corresponded to smoothing values propor- tional to the detection window aspect ratio. Experiments show that the distribution of detection scores at any scale level is elongated, with stretch of the distribution being proportional to the aspect ratio of detection window. We find that the same conclusion holds for other object classes

5.6 Conclusions 69

such as cars, motorbikes, horses. It is unclear why this is so. For cars we suspect that the reason is that only one detector was learned for all 360◦ views. The training images were normalised with respect to their height. Thus the variation along the width was much greater than the vari- ation in height. The classifier adapted to these variations and thus is more robust to lateral shifts then to vertical shifts. Similar reasoning holds for motorbikes and bicycles, where the probabil- ity of having in-between views is high compared to exactly side or front/rear views. However it is unclear why the distribution for the person class is elongated. In Section 4.3.5 we found that vertical (2×1 cell) blocks outperform horizontal (1×2 cell) blocks. This implies the vertical encoding is more relevant and it may be that the classifier learned more variation for vertical scenarios. The fact that long vertical edges dominate for the human class may also be relevant.

Giacomo Balla, Dynamism of a Dog on a Leash, Oil on canvas, 1912. c Albright-Knox Art Gallery, Buffalo, New York.

6

Oriented Histograms of Flow and

Appearance for Detecting People in Videos

Motion information can be very characteristic, particularly for animal and human motions. Psy- chological experiments show that humans can easily recognise human actions from the motions of just a few dots placed at the joints of the actors1_{. Such experiments were popularised by Jo-} hansson [1973], who placed light emitters on their joints and filmed actors performing various activities like walking, jogging, dancing in the dark. Thus it is natural to ask: Can the detector performance be improved by including motion cues2_.

The goal of this chapter is to exploit motion cues to improve our person detector’s perfor- mance for films and videos. Detecting people in video sequences introduces new problems. Besides the challenges already mentioned for static images such as variations in pose, appear- ance, clothing, illumination and background clutter, the detector has to handle the motion of the subject, the camera and the independent objects in the background. The main challenge is thus to find a set of features that characterise human motion well, while remaining resistant to camera and background motion.

This chapter introduces such a motion-based feature encoding scheme for the detection of standing and moving people in videos with possibly moving cameras and backgrounds. It presents features based on oriented histograms of various kinds of local differences or differen- tials of optical flow. We evaluate these features both independently and in combination with the static HOG appearance descriptors of Chapter 4. The new descriptors are designed to capture either motion boundaries or the relative motion of different limbs. The experiments show that they characterise human motion well while resisting background motions.

We start with a discussion of motion compensation in Sect. 6.1 and propose differentials of dense optical flow for this. Section 6.2 presents details of the proposed motion codings, which can broadly be divided in two categories: coding of motion boundaries and internal/relative dynamics. The section provides several variants differing mainly in how the spatial differen- tials are computed. A description of the different optical flow methods that we have evaluated is presented in Sect. 6.3. Our experiments show that using dense subpixel flow is essential for good performance, however noise is better than bias so heavily regularised flow is not required. Simple pixel-by-pixel multi-scale flow estimation work well. Section 6.4 discusses optimal pa- rameter values for each scheme. Section 6.5 compares the proposed motion descriptors and studies the effect of representation choices on the performance. It shows that orientated his- 1_{There are other interesting observations as well. Human beings need only a fifth of a second to recognise}

the action [Johansson 1973]. If the actor is known to the viewer, the viewer is often able to recognise her or him [Cutting and Kozlowski 1977] or in the worst case determine his or her gender [Kozlowski and Cutting 1977, 1978, Cutting and Kozlowski 1978].

2 _{Even in computer vision, Johansson’s movies of dots in biological motion have been an inspiration, e.g.}

74 6 Oriented Histograms of Flow and Appearance for Detecting People in Videos

tograms of differential optical flow give the best overall detection performance and that the proposed motion descriptors reduces the false alarm rate by an order of magnitude in images with movement while maintaining the performance of the original static HOG method in sta- tionary images. The complete motion HOG encoding algorithm is presented in Section 6.6. We conclude with a discussion in Sect. 6.7.