In the Tetris task (Jermann et al., 2010), we analyzed how the group compositions affect eye- movements patterns. We first pointed out statistical differences in their patterns of actions and eye-movements and then, we built machine learning classifiers to do predictions about group compositions using the same features. We also compared our prediction abilities when using only gaze features, only actions features or both to find out which aspect is the most predictive.
Method
Players were split in two groups, novices and experts, which resulted in four possible group compositions, EE, EN, NE and NN. Note that the first letter represents the player under consideration, while the second letter corresponds to its partner. Hence, EN represents an expert player playing with a novice, while NE represents a novice playing with an expert. We computed several gaze features which are the percentage of fixation time on the different parts of the game (self’s piece, other’s piece and stack of already fallen pieces) as well as simple properties of fixations and saccades: average fixation duration, average saccade length and the ratios of horizontal and vertical saccades. Action features are composed of the frequency of the different types of basic actions (translations, rotations, drops) done by each player and higher-level features representing how fast and how direct players played their piece. These higher-level features are the horizontal directness, which measures the amount of additional useless horizontal movements that have been done to place the piece correctly, and the vertical acceleration, which measures whether and how much players accelerated the fall of their piece.
Chapter 5. Dual eye-tracking experiments Results
We first present results from a principal component analysis which allowed us to reduce the features space in two dimensions by still retaining almost half of the variance and to understand what variables are affected by the group composition. In a second part, we show results of classification using machine learning algorithms on the same features.
Principal Component Analysis We conducted a Principal Component Analysis (PCA) to investigate the structure of the various gaze and action features. The analysis included five gaze related variables as well as three action related features. We retain two dimensions with eigenvalues higher than 1 for the presentation of the results. The first dimension accounts for 23% of the total variance, the second dimension accounts for 20%. The correlations between the initial variables and the orthogonal dimensions of the PCA are represented in figure 5.9. The correlational structure uncovers two rather independent aspects of the behavior. We use the mapping of group compositions variables onto the two components to guide the interpretation of the dimensions (see figure 5.10).
The first dimension reflects more the individual style of play and could be seen as piece control versus solution search. Two gaze features are strongly correlated with the first dimension. It represents the opposition between fixations on the players’ own piece (self ) and fixations on the contour. The players fixate their own piece when it first appears as well as when they “drive” it through rotations and translations down towards its final position. They fixate the contour to search for a “good” spot to places their piece, as is suggested by the correlation of the ratio of horizontal saccades with the dimension. In figure 5.10, this dimension opposes novices (NN) and experts (EN and EE). Interestingly, NE players are situated to the left of NN players, indicating that when novices play along with an expert, they fixate their own piece less than when they play with another novice.
The second dimension has a more collaborative aspect and represents speed versus coordina- tion. Vertical acceleration has the strongest correlation with the second dimension. Horizontal directness also correlates positively with the dimension, indicating that few superfluous trans- lations are done. Rotations and fixations on the partner’s piece correlate negatively with the dimension, indicating a slower, but more collaborative pace of play. In figure 5.10 experts (EE and EN) are positioned higher on this dimension than novices (NN and NE), although it appears that EN experts slow down because they play with novices.
Automatic predictions We then experimented with the prediction of the roles of the player and their contexts using both generative Gaussian Mixture Model (GMM)5and discriminative 5GMM (Bishop, 1995) is a parametric probability density estimation technique which has been successfully
Figure 5.9: Correlation circle for the two first dimensions. The arrows represent the correlation between original variables and the dimensions of the PCA.
Support Vector Machines (SVM)6. We present here the classification results using these two algorithms because they appeared to be the most effective algorithms in this situation com- pared to others that we tested, such as HMM or binary decision tree. In our experiments, each group composition category (i.e. NN, NE, EN and EE) is represented by a GMM and is referred to as model. To test data, the likelihood of data is computed across all these GMMs, and the one providing the largest log likelihood is found as the recognized group composition. For support vector machines, as they are originally binary classifiers, we used Max-win strategy for multi-class recognition.
Different sets of features were used for the recognition of group composition: action-based features, gaze-based features, and their combination. The whole data for each group com-
6SVM (Cristianini and Shawe-Taylor, 1999) belongs to the class of maximum margin based discriminative
classifiers. They perform pattern recognition between two classes by finding a decision surface that has maximum distance to the closest points in the training set which are termed as support vectors.
Chapter 5. Dual eye-tracking experiments
Figure 5.10: Centroids of the group compositions variables in the space defined by the two first principal components.
position was divided by five folds, and five-fold cross validation was used for the evaluation. Table 5.3 summarizes the recognition performance using different classifiers and different feature sets, in terms of prediction accuracy (% correct predictions) and Kappa statistic. Gaze- based feature sets performs slightly better than action-based feature sets. This demonstrates that gaze-based features are more important for classifying different group compositions. Combining the two features sets significantly improves the recognition performance, which shows that the information from the two feature sets are complementary. It can be also found that SVM performs better than GMM. This might be explained by the fact that SVM is a more discriminative model than GMM which is based on the estimation of the probability density function (pdf ) statistically7.
Table 5.3: Recognition results using different features in terms of correct recognition rate (CRR) and Kappa coefficient (the higher the better).
Action-based Gaze-based both
GMM Accuracy [%] 51.34 56.62 70.84 Kappa 0.34 0.42 0.63 SVM Accuracy [%] 54.45 60.12 75.28 Kappa 0.39 0.49 0.68 Discussion
Results of the Principal Component Analysis show that players adapt their behavior and gazes to the social context of interaction. A possible explanation is that experts tutor novices by telling them where and how to place their piece on the stack, thereby showing the novices the tricks of the trade. Another explanation is that experts feel responsible for avoiding collisions or solving conflicts. As a consequence, they play slower and monitor their partner’s piece more than if they interacted with another expert. In parallel, novices who interact with experts look less at their own piece and monitor the contour more than if they interacted with novices. Moreover, we have shown that these differences allow us to computationally detect the group composition of the players. We could thus envisage the design of adaptive gaze awareness tools. For instance, experts could see whether their novice partner looks at the place they are referring to. The expert could point the novice to the correct place by gazing and verbally signaling a deictic reference (e.g. saying "here").
5.5 Interaction episodes
We now report analyses at the micro-level, i.e. that use of dual-gaze features for the charac- terization of specific interaction episodes which are relevant for socio-cognitive processes of collaboration. Such episodes include explanations, elaborations, conflict resolution or mutual regulation, which are all known to play an important role in collaboration.