In order to illustrate the issues presented in section 4.3.4 about the use cross-recurrence for gaze data in collaborative tasks, and to show the effectiveness of the proposed correction meth- ods, we did some experiments by generating simulated data. More specifically, we showed that the zone based coupling measure we propose above is an indicator of the actual coupling between viewers that is relatively insensitive to the macro-structure of the observation se- quences. We consider a simple situation in which we have a stimulus with N objects, possibly grouped by zone. The generated data are uniformly distributed over all objects, or over all objects of a zone. In addition, an artificial coupling with fixed probability is inserted in the data, i.e. some data points of one stream are forced to the value of the other stream some lag before. We also vary the type of collaboration (versus cooperation) and the possibility of revisiting a zone.
Parameters Four parameters determine the high level features of the cross-recurrence plot. • Number of zones.
• Number of objects in each zone.
• Sequentiality of zones indicates whether viewers visit zones only once or whether they return to visit them on several occasions.
Figure 4.9: Cross-recurrence plots for six scenarios. Numbers refer to the scenario ID in table 4.1
• Synchronicity indicates whether the two viewers visit zones in the same order.
Four parameters determine the type of coupling that can be generated.
• Level of coupling determines the probability for a fixation of one viewer to be identical to a previous fixation of the other viewer.
• Mean lag of coupling determines the average lag at which coupling occurs.
• Standard deviation of coupling lag determines variations of the coupling lag, this latter being normally distributed.
• Symmetry of coupling determines whether subjects are equally likely to follow each other’s gaze or whether one subject is leading the other.
Scenarios We designed six scenarios to illustrate the effect of the number of zones, the sequentiality as well as the synchronicity of exploration (see table 4.1). For the coupling parameters, only the level of coupling was varied systematically from 0 to 1 by increments of 0.1. The lag of coupling was held constant for all simulations at 30 with a standard deviation of 5.
Results The macro-structure of recurrence plots (figure 4.9) nicely reflects the sequentiality and synchronism of exploration. Off diagonal squares correspond to viewers returning on previously visited zones. This effect is absent when only one zone is visited (scenarios 1 and 2).
Chapter 4. Dual eye-tracking methodology
ID Scenario Z Objects Seq Sync
1 uniform-10 1 10 - -
2 uniform-70 1 70 - -
3 sequential-sync 4 20,10,40,20 yes yes 4 return-sync 4 20,10,40,20 no yes 5 sequential-async 4 20,10,40,20 yes no 6 return-async 4 20,10,40,20 no no
Table 4.1: Scenario parameters used in the simulation. Z stands for the number of zones, Objects refers to the size of the zones, Seq indicates whether the exploration was sequential (one pass) or with returns on previously visited zones, Sync indicates whether the sequence of zones is the same for both viewers.
Figure 4.10: Diagonal recurrence rates for six scenarios. Subplots correspond to the recurrence plots in figure 4.9. Vertical scale is identical for all subplots.
The shade of gray is proportional to the baseline of the zone. The more objects are available for inspection, the less chance for recurrence at high lags, the lighter the shade of the plot. The recurrence rates obtained from the simulated scenarios are depicted in figure 4.10. In all panels, a central recurrence peak is clearly visible in the graphs. This peak corresponds to the artificial coupling of the viewers that we have introduced in the simulation. The flat baseline in panels a and b are typical of the ideal case with only one zone of objects. The combination of zones containing various numbers of objects (and therefore various baselines) results in a sloped baseline best visible in panels c and d. The off diagonal rectangular shapes in the recurrence plots 5 and 6 (figure 4.9) create “bumps” in the recurrence rates represented in panels e and f.
Figure 4.11: Decomposition of the recurrence into zones. Subplots correspond to the recur- rence plots in figure 4.9
When diagonal recurrence rates are computed on a zone by zone basis, each zone features its own baseline and peak (see figure 4.11) and resembles the simple case with only one zone. The overall recurrence graph is a weighted sum of the zone recurrence graphs (proportional to the time spent in the zone). From these subgraphs, we apply the formula 4.6 to obtain the coupling.
Figure 4.12: Recurrence rate and normalized coupling given a theoretical coupling from 0 to 1. To estimate whether the measured coupling indicator is a good indicator for the actual cou- pling between viewers, we plotted values of the measured coupling as well as the raw recur- rence rate against the values of actual coupling. Figure 4.12 shows that the measured coupling (right panel) is partially neutralizing the high-level effects of sequentiality and synchronicity of
Chapter 4. Dual eye-tracking methodology
the data streams and that it is a good estimator of the actual coupling between the viewers. On the contrary, we see that the raw recurrence rates on the left panel of the figure are sensitive to baseline variations due to the macro-structure of the recurrence plots and thus do not reflect the actual short-time coupling.
4.3.6 Discussion
We have seen that cross-recurrence is a very interesting indicator to study relationships between eye-movements of people interacting verbally. Indeed, Richardson and Dale (2005) have highlighted the existence of a gaze coupling phenomenon that is related to the type and quality of the discussion. However, we have also seen that cross-recurrence analysis may be affected by other phenomena than the short-time coupling due to dialogue. Indeed, in real collaboration tasks, there are several issues that can affect cross-recurrence analysis. The source of the various problems is generally the fact that when collaborators look at a smaller area, or in an area which contains less objects, then they have a greater chance of being recurrent than with a bigger, or denser, area. This effect may be present in different ways.
First, when comparing different pairs of collaborators, it may be the case that some pairs worked by focusing on a smaller part of the stimulus than other pairs. This makes the general level of recurrence not comparable between pairs. A simple correction for this issue consists in computing a random recurrence level by shuffling the gazes of the collaborators, or by computing the average recurrence level for high lag values, and then by subtracting this random level from all recurrence rates.
Second, depending on the task, collaborators may explore the stimulus by zones which results in recurrence peaks for lags corresponding to differences between moments when subjects explored the same zone. Here, we propose a more general correction that would take into account the zones and compute the recurrence separately in each zone. The resulting recurrences per zone are then combined by taking into account the size or density of the zone to correct for the random level. We have proven the effectiveness of this correction method with simulated data but it is difficult to assess how well it would work with real data. Moreover, its application to real situation is difficult because we have no good and objective way of defining the zones. A possible solution would be to define zones by applying a spatio- temporal clustering of the fixations in order to identify regions that are looked more or less uniformly for some time. Roughly, this would consist in detecting "meta-fixation", i.e. periods during which the fixations remain in a certain zone.
Finally, a last problem is that zone, or stimulus may be explored with some temporal trend, i.e. that there are more chances to explore a part of the zone at the beginning of the exploration than at the end, which increases artificially the recurrence level for small lags. This issue remains unsolved.
4.4 Discussion
In this chapter, we have presented some of challenges related to the analysis of eye-movements in collaborative situations. We have also offered some possible solutions to these issues. First, on the technical side, we showed that dual eye-tracking requires to synchronize in some way the gaze data recorded by two eye-trackers which do not have any specific synchronization mechanism. In section 4.2.2, we proposed and experimented various setups to accomplish this and we ended up with a very satisfactory solution which makes both eye-trackers to use the same timer.
The second challenge concerned the analysis of dual gaze patterns, i.e. the analysis of the relationships between eye-movements of two collaborators. We focused on a specific type of analysis, first used by Richardson and Dale (2005), which is cross-recurrence analysis. Indeed, such analyses have been used to show and measure the coupling between eye-movements of two people discussing together (see section 4.3.3). In section 4.3.4, we present and discuss this analysis in details and we show the difficulty of using it in real complex collaborative situations. We finally propose different possible solutions to overcome effects which are not due to the phenomenon of interest. The most promising solution consists in computing a zone-based cross-recurrence which permits us to cancel out the effects of an exploration of the stimulus by zones. Finally, we present some simulation experiments which illustrate these difficulties and show the effectiveness of the correction proposed (see section 4.3.5).
The second part of this thesis is dedicated to the presentation of experimental studies using dual eye-tracking. These studies have been conducted to identify patterns in the collaborators eye-movements which reflect the collaborative activity. This chapter present four exploratory experiments that have been conducted in this goal and which cover a wide range of tasks ranging from very conceptual activities such as concept-map building to sensory-motor tasks such as collaborative Tetris game. They illustrate how we progressively gained experience in dual eye-tracking. The next chapter will present the main experiment which has been designed from the experience gained during this exploratory phase.
5.1 Introduction
We first present a detailed description of these different experiments. We then expose the main results grouped according to which aspect of collaboration they are related to. The two first parts concern collaboration at the macro-level, i.e. relating eye-movements to the whole interaction. More specifically, we are interested in making relationships between collaboration quality on one hand and group composition, i.e. the expertise and difference of expertise between collaborators, on the other hand. The two other aspects are related to the micro-level, which consists in the analysis of short interaction episodes. First, we are interested in finding relationships between eye-movement patterns and the type of episodes, such as conflict or explanation episodes. Secondly, we study in more detail explicit referencing, i.e. when collaborators explicitly reference visual objects.
Two types of results are presented. On the one hand, we do classical statistical analyses that relate gaze features and collaboration indicators. This allows us to get some insights on how these two things are related. On the other, hand, we sometimes managed to do actual predictions about some aspects of collaboration from eye-movements patterns using machine learning algorithms. These results show the potential of dual eye-tracking for gaze aware applications. Indeed, by being able to make automatic predictions about the ongoing collaborative process, it may be possible to provide in real-time some help or feedback to
Chapter 5. Dual eye-tracking experiments
the collaborators to enhance their interaction. However, this type of results can difficulty be interpreted in cognitive terms.