Even so, the stimuli themselves can be ambiguous and many external, and hard to control for, factors influence the ratings. Examples include mood, time of day, familiarity with the stimulus, order of stimulus presentation, etc. etc. • When doing EEG experiments, it is of critical importance to do everything pos-
sible to minimize the influence of external factors. This includes having the participant in a noise-isolated separate room during the experiment, insuring a constant level of lighting and temperature, minimizing subject fatigue (by including breaks and limiting the total experiment duration). Also, it is impor- tant to make sure the experimental system is well-tested to avoid nasty surprises (crashing experiments, data loss) with the first set of participants. Finally, care should also be taken with the choice of stimulus material, in order to maximize the chances of eliciting the desired affective responses.
• In single-trial analysis, we found substantial differences in performance for dif- ferent participants. Also, training a one-fits-all classifier did not work nearly as well as subject-dependent classifiers. In addition, due to changes in external fac- tors mentioned earlier, classifiers may not generalize well over time. To achieve the best performance, affective computing platforms should therefore attempt to model the context as well as adapt to the persons using them.
• When working with video material, it is easy to identify mistakes in labelling upon inspection of the data. For instance, when testing a facial expression classifier, it is easy to notice that the labels for Action Units 12 and 45 were accidentally swapped, just by looking at the video. This is not the case for EEG signal data, which all looks pretty much the same, and the utmost care should be taken, with checks along every step of the way, to ensure no mix-ups occur.
7.3
Directions for future research
The field of affective tagging, and affective computing in general, is still very young and many questions remain unanswered. Some possible directions for future research include:
• An interesting question is to investigate how well methods such as those pre- sented here will generalize, both over time, and over participants. How do context, mood, etc. influence the performance of such methods?
• in this work, we considered each trial as a single data point. However, one may feel a wide range of emotion whilst viewing for instance a music video. It would
7.3. Directions for future research 130
be very interesting to consider the temporal variation of affective state. This would however, probably also require temporal ground truth annotation, which may be difficult to obtain without distracting participants too much.
• The modalities investigated in this work represent only a subset of possible modalities that can be used in affective computing. Modalities such as body pose, MEG, voice and non-vocal utterances, gaze, gestures, device interaction measures(e.g. typing speed), can all play in role in assessing a user’s affective state. Fusing information from all of these modalities and identifying their relative strengths and weaknesses may increase affect estimation performance substantially.
• It has been shown in previous work on EEG that task-relevant events often elicit stronger responses than task-irrelevant events. In this thesis, participants were not given specific tasks to complete. In general, giving a participant a task goes against the concept of implicit tagging, where the goal is to assign tags by observing the user without any active participation from that user. However, one can envisage several scenarios in which users give themselves a task to complete, for instance while performing work tasks, or while playing a game. In these cases, the implicit tagging paradigm can still apply. It would be interesting to investigate whether in such cases, affective state estimation for task-relevant events performs better than for task-irrelevant events.
• Whilst the methods and apparatus used in this thesis work well in a labora- tory setting, they are not directly applicable in real life. Recently, many new devices have been introduced that are unobtrusive and user-friendly, offering perspectives for use in everyday life (such as simple EEG devices with fewer (dry) electrodes and short setup times, physiological sensors embedded in ev- ery day devices, etc. etc.). Investigating if and how well methods for affective state estimation will translate when used with such devices is an interesting perspective.
Appendix A
A Real-time affective
recommendation system
A.1
Introduction
As a proof of concept, a real-time affective music video recommendation system was implemented. This was a collaborative effort within the framework of the EU FP7 project PetaMedia1. Besides the EEG analysis method described in this work, mod- ules were also created for affective state assessment based on multimedia content analysis and peripheral physiological signals. The output of all these modules was then merged into a single estimate of the user’s affect. This estimate was then used as an input to a music recommendation module, yielding real-time affective music video recommendations.
The implementation of and subsequent experiment with this real-time recommen- dation system served two main goals:
1. Design and implement a real-time affective recommendation system to demon- strate the feasibility of this approach.
2. Evaluate what the added value is of affective feedback in this content recom- mendation systems.
Using the same techniques as described in Chapter 4, a dataset containing 300 music videos was collected. These videos were then all rated online by volunteers in terms of valence and arousal. An existing music recommendation system based on Last.fm taste profiles was adapted to accept affective inputs and used to deliver the recommendations.
The system operates as follows. First, the participant gives his Last.fm username. His most recent songs are retrieved from the Last.fm database and personalized clus- ters are generated for the 300-song dataset, based on the participants’ taste profile as defined by the recent songs and the average affective ratings by online volunteers. 1PetaMedia: http://www.petamedia.eu. Collaborating researchers were: Mohammad Soley-