From our valid data collection, we extracted a subset of 10 game rounds randomly chosen for validating the algorithms. In a second step, another subset of 10 game rounds has been picked up for testing the algorithms.
5.3.1.1 Ground Truth
The only medium from the data collection we could access for preparing the ground truth files were the videos of the hands. We annotated the videos of the training subset by observing when the hand stayed in a stationary position. Based on this annotation, the ground truth files were generated to match the events of the videos and to indicate when the hand was in a stationary position. To align data correctly, we defined a reference duration (the starting time was the start of the Leap Motion for the concerned round, the ending time was the last tap performed at the concerned round).
From the observation made during the data collection and the videos, we could clearly observe different behaviours among participants. On of the most striking personal fea- tures were related to how the hands are kept above the tablet and the change of hands during the interaction. Figure 5.3 illustrates the first difference: some participants pre- ferred keeping their hand away from the tablet as much as possible, hence moving above the tablet’s display only when a tap was required (Figure 5.3a), whereas others kept their hand above the surface, either in a relatively still position to move only when tapping or “following” the gaze while skimming the tablet’s content (Figure 5.3b. This trends were not systematical: some participants were showing both type of trends during the interaction. A visual classification based on the video we took during the data collection indicates that on average during the interaction with the tablet in this context of playing Memory Game, most participants maintained they hand over the display (69 % of them, 81 participants) while more than a quarter of them (29 %, 34 participants) did not have a specific pattern and sometimes kept the hand above the display and sometimes retracted it back closer to their body. Only 2 participants mainly retracted close to their body during most of the interaction.
The other personal feature we observed concerned the switch of hands during interaction. Despite the fact participants reported a dominant hand before the interaction started, we
5. Correlation between Gaze and Stationary Hand Event
noticed that all participants did not necessarily interact with the tablet solely with that dominant hand: some participants were more likely to switch hands during interaction. We associate this behaviour with two factors: fatigue and/or ease of reach (less effort is required to reach the left side of the tablet with the left hand and the right side of the tablet with the right hand). Video analysis showed than on average during interaction, participants solely use their dominant hand (90 %, 105 participants), and few users preferred using both hands: 6 participants (about 5 %) used both hands to reach each side of the tablet, while 6 others used one hand a time alternatively.
(a) The hand is returning to a base position near the body between taps.
(b) The hand is staying near the tablet screen and can follow the gaze path. Figure 5.3: Screenshots of different hand moving trend sequences during interaction.
When observing the personal feature on a game level basis, we noticed that often, partic- ipants changed their interaction pattern during playing. For example, some participants started playing the first level with keeping a close distance between their hand and the tablet, then started to retract the hand close to their body for the last level - and some participants did the exact opposite. The conclusions from this change of behaviour are difficult to draw, because it is not clear which factors influences this change. Fatigue and/or mental load are two candidates but there is no evaluation possible to assess which of them are the most influential on the change of behaviour. From our population sample, 26 % of the participants (30) changed their personal pattern (as keeping the hand above the surface of the tablet or retracting it) and 13 % of them (15) changed hand pattern for the left/right hand usage during the whole interaction (for instance starting with both hands then only one hand).
5. Correlation between Gaze and Stationary Hand Event
5.3.1.2 Hovers and Dwell Definitions
The nature of such data expectingly results in a highly skewed classification, as suggested by Table 5.1: during interaction the hand was most of the time in movement rather than stationary (from our validation subset, stationary hand events lasted between 12 % and 48 % of the overall duration of the round).
In order to detect the type of the stationary hand event (hover or dwell), we evaluate where the hand was targeting at, based on the hand position retrieved from the Leap Motion, and on the dominant eye position retrieved from the eye tracker. During the duration of an event, we compute the projection of the hand from the eye, as illustrated by Figure 5.4. From the Leap Motion, we select the finger tip point that was the closest to the tablet’s top (lowest Y-value in the tablet coordinates) in order to compute the projection. If the projection is on the tablet, the stationary hand event is considered as hover, otherwise it is labelled as dwell.
Figure 5.4: Projection (P) of the hand on the tablet from the user’s eye perspective.