Valoración de escala de Conners

In this work, two main research hypotheses have been explored. The first is that, event- based game-data representation can be used to predict player behaviour with supervised learning and can provide a better performance compared to random- guess and other state-of-the-art methods in a wide range of games. This study aims to investigate the ability of the proposedevent-frequency-based data representation.

The other main hypothesis is that,the disengagement-over-varying-dates labelling method can be used to make use of all data samples while maintaining an ap- proximately balanced dataset, and that parameters optimised for balancing can be used as indicators of a game’s health. This study investigates the new labelling method (named disengagement over varying dates), which is designed to give an alternative way to deal with the possible bias and lack of data problems in some datasets caused by the widely used churn-labelling method.

8.1.2 Contribution Summary

Several contributions have been made over the course of this study. In this section, all the contributions made by this work are summarised and reviewed. Discussions are given on how well the hypotheses were supported based on the results of the experiments.

Event Frequency-based data Representation

In this work, a review has been given of several studies that have attempted to predict player behaviours using data-mining approaches in games. One of the common limita- tions of these works lies in their inability to be generic: The proposed methods cannot readily be migrated to different products. Aimed at working out a possible solution to the limitation from the selection of data representation, this work first analysed the

reasons for the low generality of existing works (in Section 4.5.1). It was claimed that the generality of a data representation often comes from two aspects: game-specific and availability. Some existing works have worked to solve thegame-specific issues by relying only on some session-based features; however, the session related information is not always collected in games, and it often may not be accurate enough. To cope with these two problems at the same time, as the first main contribution, this work takes the counts of any collected events (player behaviours or system events) as the data representation. In this way, because only counts of events are used instead of their actual meanings, there is no need to understand the true meaning of the events before the method can be applied. For the same reason, this method is not restricted to any specific game, and the same implementation of the data representation can be easily extended to work for different games. Besides, since this approach takes any events that are collected in games, it is able to maximise the use of the data and reduce the chances of encountering availability issues. To investigate its performance, the first hypothesis in Section 8.1.1 was proposed.

Player First-purchase behaviour Prediction

To verity this hypothesis, this work uses event-frequency-based data representation to predict players’ first-purchase behaviours in three different commercial games. In the relevant experiments, event-frequency-based data representation exhibited its general- ity, as it shows robust performance across all three different genres of games without any special pre-processing and is able to provide promising predictive performance which is significantly better than random guesses. This gives positive support to the hypothesis made for this data-representation method.

Disengagement-labelling method

The churn-labelling method has been widely used to represent disengaging behaviours in games. It focuses on predicting disengaging actions in which players entirely stop playing a game. However, Runge et al. (2014) discuss the fact that players who have been predicted to churn can hardly be held back by simple in-game rewards. To give de- velopers an earlier chance to deal with possible churning users, a new labelling method called ‘disengagement’ was introduced that focuses instead on players’ disengaging trends.

Player Disengagement/Churn Behaviour Prediction

To further validate the hypothesis concerning event-frequency-based data representa- tion, it has been further used to predict players’ disengaging behaviours with both the newly proposed disengagement-labelling method and the churn-labelling method used in the work by Runge et al. (2014). During the experiments, the game-specific data representation introduced in the work by Runge et al. (2014) was also added for com- parison. Experiments predicting the disengagement (labelled) behaviours show that event-frequency-based data representation exhibits good generality and can achieve competitive performance. Combined with the experiments for predicting the first- purchase behaviours, both experiments offer positive support to the first hypothesis made in this study. However, during the experiment in which event-frequency-based data representation was applied to predict players’ churn behaviours, we saw that clas- sifiers trained with the event frequency-based data representation can in most cases still achieve competitive performance compared with those trained with game-specific data representations. This hypothesis was not well supported, as there are cases in which

CHAPTER 8. CONCLUSION 135

the classifiers trained with the event frequency-based data representation behaved like random classifiers. As discussed in Section 6.4, this might be caused by the limited number of examples available for training, as classifiers (especially higher-dimensional ones) can hardly be well trained under this situation.

A Review of Two Popular Class-balancing Methods

A general problem in data-mining research is that the class distribution can be biased if the labelling method distributes training examples into different classes by some fixed conditions. In this work, the general causes of a biased classification problem were dis- cussed and two widely used methods–random undersampling and SMOTE (Synthetic Minority Over-sampling Technique)–were introduced. In Chapter 7, statistical com- parisons were given to show whether random undersampling and SMOTE would help reduce class bias such that the performance of the classifiers can be improved. Based on experiments, summaries were given for these two methods, and it was suggested that random undersampling is more likely to achieve better performance when the number of data samples is large enough, whereas SMOTE is better able to be used with smaller datasets. If the data sample is too small, none of the methods help significantly. Disengagement-over-varying-dates labelling method

Two important problems faced by some experiments in this study are lack of data samples and biased class distribution. As discussed in sections 6.5 and 7.1, both of the issues can be found when the churn-labelling method is applied, as the predictive methods and classifiers trained in this situation cannot behave well. To provide more reliable classifiers for predicting player behaviours, the second main contribution of this work is to propose an alternative labelling method that also reflects players’ disengaging behaviours but is able to use all data samples while maintaining an approximately balanced dataset. In addition, parameters optimised for balancing in this method can be used as indicators of a game’s health. To investigate its effects, the second hypothesis in Section 8.1.1 was proposed. Experiments in Section A show that this labelling method can help balance the class distribution and provide a larger dataset in all three commercial games for predicting players’ disengagement behaviours. Classifiers trained under this situation were able to behave significantly better than a random classifier. This result supports the hypothesis proposed and suggests that this labelling method can be an alternative when reliable classifiers cannot be reached for predicting the original churn behaviours.