Throughout the course of the forthcoming experimental work, two different eye tracking systems were used. During initial development, the evaluation presented later in this chapter, and the experiment presented in Chapter4, the MobileEye eye tracker from Applied Science Laboratories (ASL) [Lab10] was used. The MobileEye eye tracker is head-mounted and features two cameras: one monitoring the right eye (eye camera) and one recording the scene as viewed from the perspective of the wearer (scene camera). Video feeds from both cameras are interleaved and recorded using a battery-operated digital video camera recorder (DVCR). The tapes have a typical recording duration of 60 minutes. Together with recording the video streams, DVCR also outputs gaze tracking data, in real-time, in machine- readable form on an IEEE 1394 (FireWire) cable connected to the capture PC. The MobileEye eye tracker operates with the dark-pupil principle based on the corneal-reflection of three infrared light- emitting diodes (LEDs) mounted alongside the eye camera. Image analysis determining positions of the pupil and the reflections produced by the LEDs on the pupil/corneal boundary are used to determine eyeball rotation. This eye-in-head direction is mapped to x/y coordinates with respect to the scene camera video, on which the coordinate is also superimposed, thus indicating where the wearer is looking. The coordinates are also made available on a serial port for external processing. The MobileEye eye tracker outputs data at a rate of 30 Hz (33 ms). The system was mounted on the CAVE’s shutter glasses, as illustrated in Figure3.3.
During the early stages of this research, several technical and practical issues with the MobileEye eye tracker, when operating in combination (‘through’) the CAVE’s shutter glasses as shown in Figure 3.3, were observed: the low-level of ambient lighting typical in CAVE systems severely limited tracking ability; the eye tracker failed to robustly track people with light (blue or green) irises; contact lenses caused distortion in the corneal reflection points as viewed by eye camera; and the always-connected DVCR hindered the wearer’s physical movement.
3.1. EyeCVE System Overview 86
Figure 3.3: ASL MobileEye eye tracker, and also head tracker, mounted on CAVE’s shutter glasses to combine head and eye tracking. The MobileEye system is used in the initial evaluation of EyeCVE appearing in this section, and in the conversational experiment presented in Chapter4.
Figure 3.4: Arrington Research ViewPoint EyeTracker eye tracker mounted on the CAVE’s shutter glasses (left) and the WALL’s empty frame (right). The CAVE setup is used in the object-focused ex- periment in Chapter5, and for data capture in the gaze and eyelid simulation work in Chapter7. The WALL setup is used in the truth and deception experiment in Chapter6.
EyeTrackersystem by Arrington Research [Res10a]. The ViewPoint eye tracker provided a more robust and lightweight solution than the MobileEye eye tracker, and also provided a greater update rate of 60 Hz (16 ms). Rather than using corneal reflection to track the eyes, the ViewPoint eye tracker uses infrared LEDs to illuminate the eye. The ViewPoint software then processes the eye camera’s video- stream to identify the location of the pupil. The pupil’s position is used to calculate x/y gaze position based on prior calibration data, and this gaze position is superimposed onto the output from the scene camera, which is streamed directly to the connected PC rather than recorded on a DVCR. Upon request, Arrington Research provided 160◦wide-angle lenses for the scene cameras. This FOV approaches that of humans, which is considered to be around 180◦[Cos95]. A critical feature of the ViewPoint eye tracker is its ability to output a detailed range of oculesic data including saccade velocity, fixation duration, pupil aspect ratio, and pupil size. This facilitated further integration of oculesic behaviour beyond just gaze, including blinks and pupil dilation. The ViewPoint eye tracker was used in all of the forthcoming experiments presented in this thesis, apart from those stated above. The system, mounted on the CAVE’s shutter glasses, and also mounted on the WALL’s lenses-free frame is illustrated in Figure3.4.
The following three subsections detail the calibration procedures that are required before a user enters the shared VE of EyeCVE. The gaze calibration procedure is common to both the MobileEye and
3.1. EyeCVE System Overview 87 ViewPoint eye trackers, while the latter calibration pair of blink and pupil dilation are only relevant to the ViewPoint system.
Gaze Calibration
In order to determine precisely what a user is looking at, some prior calibration procedure is required. During this process, the wearer looks at a series of points while the eye tracker records the value that corresponds to each eye position. An accurate and reliable calibration is essential for obtaining valid and repeatable eye movement data. Before calibration in the ViewPoint system, the pupil, as viewed by the eye camera, must be isolated with appropriate threshold settings. In the MobileEye system, the corneal reflections, in addition to the pupil, must be isolated. Both trackers start up in a coarsely-calibrated state that provides precise timing of raw eye movements. This is sufficient for many applications that utilise relative eye movements, such as quadrant-wise “preference of looking” tasks. However, EyeCVE requires more precise gaze information in order to map to avatar gaze in a meaningful way, so further calibration is required. Raw pupil and corneal reflection locations, captured by the eye camera, do not indicate where the wearer is looking in reality, as viewed by the scene camera. Hence, the purpose of calibration is to provide a mapping between the raw data and the gaze direction in the scene.
Figure 3.5illustrates the graphical user interface (GUI) for the ViewPoint eye tracker. During calibration in both the MobileEye and ViewPoint systems, a number of points are sequentially presented to the user, who is instructed to foveate each point in turn and keeping their head still for the duration. The software then determines appropriate coefficients for the mapping. Completing the initial gaze calibration, the accuracy of a calibration should be tested by asking the user to look at particular locations in space, and verifying that the location superimposed on the scene camera video matches. The final stage of calibration is then performed in the 3D VE of EyeCVE. This process is presented in Section 3.1.4covering the avatar subsystem, and provides a further mapping to enable avatar gaze replication. Gaze was tracked, displayed, and analysed in all three telecommunication experiments presented in Chapters4,5, and6.
Blink Calibration
The ViewPoint eye tracker is capable of outputting a range of oculesic data. This includes pupil aspect ratio: a dimensionless value, with 1.0 indicating a perfect circle. When humans blink, the eyelid descends to cover the eye before rising again to approximately its original position. Correspondingly, as viewed by the eye tracker’s eye camera, the elliptical fit to the pupil becomes increasingly flat before it disappears for a brief time, and subsequently reappears and returns to its circular outline following the blink. This characteristic change in the aspect ratio of the elliptical fit to the pupil can be used to detect blinks, which can be classified as the pupil aspect ratio crossing below a particular threshold. Rather than dynamically monitor eyelid position during a blink, which would involve significant image processing analysis, the pupil aspect ratio signal is then used to initiate a simulation of human blink dynamics, which is the focus of Section7.2. Figure3.6shows views captured from the eye camera as the pupil aspect ratio crosses below the threshold specifying a blink. Experience with the system established that a threshold of around 0.6 provides robust blink detection in the majority of wearers. Blinking was tracked, represented, and
3.1. EyeCVE System Overview 88
Figure 3.5: ViewPoint GUI. The calibration procedure maps the eye position as observed by the eye camera to the gaze direction superimposed on the scene camera view.
Figure 3.6: Views captured from the ViewPoint eye camera during the down-phase of a blink. From left to right, the tracked pupil aspect ratio gradually flattens until a threshold is met, signalling a blink.
analysed in the truth and deception experiments presented in Chapter6.
Pupil Dilation Calibration
The primary input to gaze calculation is the position of the pupil within the eye camera’s stationary view. Along with position, for gaze, and aspect ratio, for blinks, pupil size is also monitored by the ViewPoint eye tracker. The ViewPoint eye tracker delivers x and y values, normalized with respect to the eye camera’s video dimensions which have a 4:3 aspect ratio, to represent pupil size. Thus, the x- and y-values are anisotropic, and the y-value must be are initially rescaled by 0.75 (due to the difference in aspect ratio) before computing the area of the pupil ellipse using the formula: area = πXY.
The above calculation defines a user’s current pupil size, but does not provide information revealing the extent of constriction or dilation with regards to a particular individual’s natural pupil size range. In order to position the measured pupil size in this meaningful context, a wearer’s natural range, between extreme constriction and extreme dilation, must first be established. These responses are elicited by triggering the pupillary light reflex. First, the wearer is placed in a high-luminance environment to trig-
3.1. EyeCVE System Overview 89
Figure 3.7: Views captured from the ViewPoint eye camera during pupil size calibration. Left: extreme constriction in high luminance lighting.Right: extreme dilation in darkness.
Figure 3.8: Basic avatar model. Avatar is used in the conversation and object-focused experiments in Chapters4and5.
ger constriction. This is followed by placing the wearer in dark surroundings, triggering full dilation. During this calibration procedure, wearers are instructed to maintain a forward gaze direction, and to keep blinking to a minimum. During exposure to both light and dark surroundings, the wearer’s pupil size is monitored for 30 seconds. The minimum (in light) and maximum (in darkness) pupil sizes are used to establish the individual’s range. Hence, following this calibration procedure, practical minimum, maximum, and current pupil sizes are known. These measures may be used to determine the current dilated state of a wearer’s pupil, which acts as the primary input to the avatar pupil animation system, detailed in the following section. Figure3.7shows views captured from the eye camera during pupil di- lation calibration. Pupil size is tracked, represented, and analysed in the truth and deception experiments presented in Chapter6.