We demonstrated that the contributions of geometric constraints, navigation information and depth-based skin segmentation to face detection are remarkable. Based on their com- bination, mobile robots are able to localize human faces in real time, and interact with humans in a more reliable way. In our future work, we focus on two promising direc- tions that are task-based face detection, and face tracking using navigation information. By combining Microsoft Kinect color and depth images with robot navigation infor- mation, we may compute a face probability distribution of every familiar person in an indoor environment. For instance, we could set a specific schedule for a mobile robot to visit us in our office at 6 AM every working day. By using the available face probability distribution and the map of our office building, we could make the robot to ignore faces
3.6 Summary
(a) (b) (c)
(d) (e) (f)
(g) (h) (i)
Figure 3.5: Result of our face detection in different cases. 3.5b, 3.5c: A group of people standing at roughly the same distance; 3.5a, 3.5d, 3.5e: Several people standing at dif- ferent distances to the robot; 3.5g, 3.5h, 3.5i: People standing near a wall; 3.5e: People with different skin color; 3.5f: People at far distances.
Chapter 3 Real Time Face Detection Using RGB-D Images
on the way to our office and detect only faces in the region that occur very often every day, such as our working places, and identify the faces. In the case when the robot can not find our faces, the searching regions will be extended to the positions of lower face probability. We also develop a second direction for a mobile robot to utilize the available face probability distribution to improve the accuracy of the 3D particle filter algorithm for tracking faces. When human moving directions are repetitive, a robot can compute prior probabilities to predict the next face positions in 3D space. The results of a 3D particle filter should be more accurate because of a known face probability distribution.
Chapter 4
Face Tracking and Pose Estimation
Using an Adaptive Correlation Filter
4.1 Introduction
Both face tracking and face pose estimation play key roles for human-robot interaction which can be used as essential preprocessing steps for robust face recognition or fa- cial expression recognition. A reliable face tracker provides a setting where well aligned faces in frames can be integrated over long runs, which potentially leads to more accurate face recognition. In fact, the accuracy of tracking directly impacts the ability to recog- nize human faces. As a result, an efficient face tracker, which significantly improves robustness against abrupt appearance changes and occlusions, is crucial for successful video-based face recognition. Similarly, a robust algorithm for face pose estimation can be also beneficial to face recognition systems, because one of the key challenges for current face recognition techniques is how to handle pose variations. Accurately localiz- ing the face and estimating its pose are essential steps for further analysis, such as face recognition.
Despite recent progress, tracking faces in uncontrolled environments still remains a challenging task because the face as well as the background changes quickly over time and the face often moves through different illumination conditions. Moreover, previous tracking methods have significant drift due to sudden changes of the face and back- ground. In this section, we propose an algorithm of tracking faces based on the com- bination of an adaptive correlation filter (Bolme et al. (2010)) and a Viola-Jones face detection (Viola and Jones (2001b)). This combination utilizes the advantages of adap- tive correlation filters to adapt to face changes of rotation, occlusion and scales as well as adapt to complex changes of background and illumination. Its computational cost is only 7 ms per frame. Furthermore, it can also remove drift effectively by detecting the face and correcting its position after a period of time. In addition, we utilize the depth information from the Microsoft Kinect camera to estimate the corresponding size of the face. Our tracker successfully runs on a mobile robot when both the humans and the robot move and rotate quickly with different angles and directions.
Chapter 4 Face Tracking and Pose Estimation Using an Adaptive Correlation Filter
(a) (b)
(c) (d)
Figure 4.1: Examples of our face tracking and pose estimation on a moving mobile robot. The white circles indicate the locations of facial features.
nificant challenges. First, the resolution of faces is very low when the humans move far away from the robot. Most existing methods are accurate for estimating poses in high-resolution face images, but their performance is much worse or they completely fail to estimate poses in low-resolution face images. Second, while both the humans and the robot move in complicated backgrounds and under different illumination conditions, facial features change very quickly and the face also changes in a variety of poses. Ad- ditionally, when the face changes by large angles of rotation, some parts of the face are visible while the rest is occluded. In order to find a robust method of face pose estimation for human-robot interaction, we track some key facial features, including the two exter- nal eye corners and the nose. These features provide geometric cues to estimate precisely the yaw angle and the roll angle of the face which are important for the improvement of uncontrolled face recognition. Similar to our method of face tracking, we combine an adaptive correlation filter and a Viola-Jones object detection to track these features which are robust to face rotation, face deformation, occlusion and complicated illumination. As a result, the face pose is also estimated efficiently based on some geometric computation among the face features, such as shown in Figure 4.1. Our feature based method of face pose estimation is shown to be robust to run on a mobile robot in uncontrolled illumi- nation and environments while our whole system is operating at a speed of 21 ms per frame.