INTERVENCIONES REQUERIDAS

To answer our first question regarding which visualizations, the users prefer the most; the results are displayed in figure 30. The ’S’, ’B’, ’A’, respectively stand for smiley, bar chart and animation figure. In our questionnaire, we asked our users to order these three visualizations based on their preferences. For example the string ”BSA” for a sample user shows that this user first prefers the bar chart , then the smiley face and lastly, the animation. As figure 23 shows, 4 users would have rather the string ”SBA”, 3 users, string ”SAB”, 3 users, string ”BAS”, and finally for each string ”BSA” and ”ABS” we have only 1 user. As it is implied from this chart, in total, 7 people tended toward the smiley face as their first choice, 4 people toward the bar chart and 1 person toward the animation. All these implications are depicted in figure 31. What would be concluded from these results is that, the order of best visualization in user’s view is the Smiley face, Bar charts and the Animated avatar. The summery of our user’s detail justifications on their selections is provided in the table 2.

Figure 30: The priorities of different visualizations. S represents Smiley, B is for Bar chart and A for Animation. The first letter represents the highest priority, the second letter, the second highest priority, and so on.

The next part of our user experiment evaluation is regarding the Emotiboard usefulness. This section has been fulfilled with a group of questions. Figure 32 shows our user’s responses to the question ”emotional feedback influences our collaboration”. In our questionnaire, to answer this question, we provided 7 different levels from disagree with level 1 to agree with level 7. So, a level 7 would indicate that the users are completely in agreement in this regard. And if we divide our users into two groups having positive or negative attitude, more than half of our users have positive bias for Emotiboard. Our next question in this part is ”emotional feedback is useful in our teamwork”.

Figure 31: User’s preferences of experienced visualizations regarding to their orders

Visualization Advantages Drawbacks Smiley face Simple, Not close to human face,

Clear feedback, Only three figures for smile, Understandable Stress and anger is not shown Bar charts Visible changes, Not understandable,

Colors help understanding Not related to human face Animated avatar Close to human face, No visibility in changes,

Better interception Small size, No feedback for stress Table 6: Variations of acoustic variables observed in relation to emotions

This question also follows the same format as was explained. What is implied from figure 33, the user’s attitude about the usefulness of emotional feedback on our teamwork is not strong. One possible reason for this result is raised from the nature of our experiment, since to make up conversation, we set up a non-competitive game. At one glance, having emotional feedback may not seem to be useful, but since with this experiment we could not cover the general teamwork setup, our conclusion regarding this question would lack the thoroughness and generalization. The other possible reason would be due to the lack of semantic understanding of the usage of the Emotiboard. In addition to these conclusions is that for the level 1 which, stands for the most negative bias, the number of people is zero. It shows that all our users have positive attitude toward installing emotional feedback in the collaborative work, even if this attitude is not extremely strong.

Figure 32: Emotional feedback influences our collaboration

Figure 33: Emotional feedback is useful in teamwork

The next interesting statement in our experiment is regarding the easiness of working with Emotiboard. As it can seen from Fig. 34, most of the evaluator believe that this system is easy to use.

Figure 34: I found this system easy to use

In addition to those particular questions, the students were asked to evaluate Emotiboard with 10 other complementary statements in terms of usefulness and ease of use. Please see Appendix-B for further detail of questions.

6.4

Conclusion

The user evaluation which is performed in within-subject design. The user evaluators were a small group of students from the physiology department of the University of Fribourg, Switzerland, who were recruited as 6 groups of 2 people in a team-work scheme, in a one hour long session for each group. We setup the remote feature Emotiboard in an environment with having two rooms separated from each other with a door while the users communicated with each other through Skype call. We used questionnaires as our inquiry method in order to gather subjective input such as preferences and ratings from the participants. The entire experiment last 12 hours.

The results show that more than half of our participants liked the system and they enjoyed working with it. Most of them think this system is easy to be used and they do not really need extra, in-depth knowledge and technical support to be able to use this software. But in contrast, as it was be seen form the graph, the users are not satisfied with the graphical representation of the emotions and also the level of consistency of overall functionality of the system.

7

Conclusions and Future works

Emotiboard is a multimodal affective tool that is available to be integrated in any kind of collab- orative environment to respond to user’s emotional states. Emotiboard consists of two parts, the emotion recognizer which the integration of speech emotion recognizer from openEAR and another tool for receiving electrodermal activity as an indicator of emotional arousal. The second core of Emotiboard is a java interface which combines all the mentioned tools, processes their data and finally represent them to it users.

In this project, two different evaluations are done. First, the user evaluation which is performed in a small group of students and in within-subject design to fulfill the requirements of this study and responds to our usability hypothesizes. In this experiment we tried to provide a setup to make the evaluators have a natural conversation with any possible emotional reactions. In this regard, we asked our users to play tabu game with each other to make up a conversation. Accordingly, the results show that the needs and interest to have such application in our communications setup is encouraging. Even though, the general user’s perspective of our system is positive but we still lack the better techniques of visualization. Our users looked for a natural emotional feedback which, in one way resembles human face features and in another way is easy to understand. Some users criticized this application because it might break people’s privacy and limitations in their communications.

Second evaluation is based on computing emotion recognition scores for the arousal and the valence dimensions using SVM with the SMO learning algorithm on SEMAINE corpus. We applied polynomial kernel with degree d=1 and the best constant c=0.005 for arousal and c=0.01 for valence among 9 fold cross-validation. The results extracted from performance evaluation is also compatible with the previous studies even though it lacks precisions, possibly because of some shortcomings in the preprocessing steps .

And finally, the ultimate idea behind this project is augmenting computers with emotional feedback specifically in our communication setup which with Emotiboard, as a leading project, has many hopes. At this phase Emotiboard is fusing two emotion modalities speech and physiological signals but there is a great potential to integrate other modalities such as emotion detection through face, gesture and eye gaze with the use of Microsoft Kinect and a better usage of the accelerometer of qSensor. The visualization part of this project meets the basic demands of emotion representation but it still requires more research and improvement to increases its productivity.

References

[1] Ashish B. Ingale, Speech Emotion Recognition. International Journal of Soft Computing and Engineering (IJSCE), Issue-1, March 2012.

[2] Thurid Vogt, Elisabeth Andre, Johannes Wagne, Automatic Recognition of Emotions from Speech. Multimedia Concepts and Applications, University of Augsburg, Augsburg, Germany [3] Daniel Neiberg, Kjell Elenius and Kornel Laskowski Emotion Recognition in Spontaneous Speech Using GMMs . Department of Speech, Music and Hearing, KTH, Stockholm, Sweden, Augsburg, Germany

[4] Thurid Vogt, Elisabeth Andre and N. Bee. EmoVoice - A framework for online recognition of emotions from voice. In Proceedings of Workshop on Perception and Interactive Technologies for Speech-Based Systems, 2008.

[5] J. Wagner, F. Lingenfelser, and E. Andre. The Social Signal Interpretation Framework (SSI) for Real Time Signal Processing and Recognitions. In Proceedings of INTERSPEECH 2011, Florence, Italy, 2011.

[6] de Rosis, F., Pelachaud, C., Poggi, I., Carofiglio, V., and de Carolis, B. From Greta’s mind to her face: modelling the dynamics of affective states in a conversational embodied agent. International Journal of Human-Computer Studies, 59:81-118 2003.

[7] Gilroy, S. W., Cavazza, M., Chaignon, R., Mkel, S.-M., Niiranen, M., Andr, E., Vogt, T., Billinghurst, M., Seichter, H., and Benayoun, M. An emotionally responsive AR art instal- lation. In Proceedings of ISMAR Workshop 2: Mixed Reality Entertainment and Art, Nara, Japan, 2007.

[8] Florian Eyben, Martin Wollmer, and Bjorn Schuller. openEAR - Introducing the Munich Open- Source Emotion and Affect Recognition Toolkit. Technical University of Munich, Institute for Human-Machine Communication Theresienstrasse 90, 80333 Munich.

[9] Bjorn Schuller, Bogdan Vlasenko, Florian Eyben, Gerhard Rigoll and Andreas Wendemuth. Acoustic Emotion Recognition: A Benchmark Comparison of Performances. Technical Uni- versity of Munich, Institute for Human-Machine Communication Theresienstrasse 90, 80333 Munich.

[10] Cowie, R. , Douglas-Cowie, E. , Tsapatsoulis, N. , Votsis, G. Emotion recognition in human- computer interaction. Signal Processing Magazine, IEEE (Volume:18 , Issue: 1 ), 2001. [11] Jerritta S, M Murugappan, R Nagarajan, Khairunizam Wan. Physiological Signals Based Hu-

man Emotion Recognition: A Review . School of Mechatronic Engineering, Universiti Malaysia Perlis (UniMAP) Campus Ulu Pauh, 02600, Arau, Perlis, Malaysia, IEEE 7th International Colloquium on Signal Processing and its Applications, 2011.

[12] http://www.media.mit.edu/galvactivator/faq.html

[13] Florian Eyben, Martin Woellmer, and Bjoern Schuller. openEAR - An introductory tutorial for the Munich open Emotion and Affect Recognition toolkit. Institute for Human-Machine Communication Technische Universitaet Muenchen (TUM) D-80333 Munich, Germany. [14] http://en.wikipedia.org/wiki/Data-visualization

[15] http://en.wikipedia.org/wiki/Data-visualization [16] http://en.wikipedia.org/wiki/Run-chart

[17] Jon D. Morris. Observations: SAM: The Self-Assessment Manikin, An Efficient Cross-Cultural Measurement Of Emotional Response. University of Florida, 1995.

[18] Hongfei Lin, Fengming Pan, Yuxuan Wang, Shoahua Lv and Shichang Sun. Affective Com- puting in E-learning. Dalian University of Technology, China, 2010.

[19] http://www.scholarpedia.org/article/Speech-emotion-analysis

[20] Ginevra Castellano, Loic Kessous, and George Caridakis. Multimodal emotion recognition from expressive faces, body gestures and speech . EU 6th Framework Programme, 2007.

[21] http://sox.sourceforge.net/ [22] http://www.cs.waikato.ac.nz/ml/weka/arff.html [23] http://www.cs.waikato.ac.nz/ml/weka/arff.html [24] http://en.wikipedia.org/wiki/Median-filter [25] http://weka.wikispaces.com/Primer [26] http://en.wikipedia.org/wiki/Repeated-measures-design

[27] Fabien Ringeval, Denis Lalanne, Andreas Sonderreger and Juergen Sauer. On the Influence of Emotional Feedback on Emotion Awareness and Gaze Behavior . Document Image and Voice Analysis group, Department of Informatics University of Fribourg, Fribourg, Switzerland. [28] Fabien Ringeval, Denis Lalanne, Andreas Sonderreger and Juergen Sauer. Introducing the

RECOLA Multimodal Corpus of Remote Collaborative and Affective Interactions.

[29] Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N. Chang, Sungbok Lee and Shrikanth S. Narayanan. IEMOCAP: Interactive emo- tional dyadic motion capture database. Speech Analysis and Interpretation Laboratory (SAIL) University of Southern California, Los Angeles, CA 90089 2007.

[30] Kanade, T., Cohn, J. F., Tian, Y. Comprehensive database for facial expression analysis. Proceedings of the Fourth IEEE International Conference on Automatic Face and Gesture Recognition 2000.

[31] Naiyu Wu, Huiping Jiang, and Guosheng Yang. Emotion Recognition Based on Physiological Signals. Information Engineering College, Minzu University of China, Beijing, 100081, China 2012.

[32] F. Honig, J. Wagner, A. Batliner, E. Noth. CLASSIFICATION OF USER STATES WITH PHYSIOLOGICAL SIGNALS: ON-LINE GENERIC FEATURES VS. SPECIALIZED FEA- TURE SETS. 17th European Signal Processing Conference, 2009.

[33] Cong Zong and Mohamed Chetouani. Hilbert-Huang transform based physiological signals anal- ysis for emotion recognition. Universite Pierre et Marie Curie.

[34] Hatice Gunes, Massimo Piccardi. Bi-modal emotion recognition from expressive face and body gestures. Computer Vision Research Group, Faculty of Information Technology, University of Technology, Sydney (UTS), 2006.

[35] Lance, B. and Marsella, S.C. Motionally Expressive Head and Body Movement During Gaze Shifts. In Proc. IVA 2007, Springer-Verlag, 2007.

[36] Brett Taylor and Regan L. Mandryk. Creating and Interpreting Abstract Visualizations of Emotion. Graphics Interface Conference, 2012.

[37] Florian Eyben, Michel Valstar, and Bjoern Schuller, Roddy Cowie, Maja Pantic AVEC 2012 - The Continous Audio/Visual Emotion Challenge. 2013.

[38] Gary McKeown, Michel Valstar, Roddy Cowie, Maja Pantic, Senior Member, and Marc Schroder The SEMAINE database: annotated multimodal records of emotionally coloured con- versations between a person and a limited agent. 2007.

[39] Florian Eyben, Martin Wollmer, and Bjorn Schuller. openSMILE the Munich open Speech and Music Interpretation by Large Space Extraction toolkit. Technical University of Munich, Institute for Human-Machine Communication Theresienstrasse 90, 80333 Munich, 2010. [40] Soroosh Mariooryad, Carlos Busso. Analysis and Compensation of the Reaction Lag of Eval-

uators in Continuous Emotional Annotations. Humaine Association Conference on Affective Computing and Intelligent Interaction (ACII), 2013 .

[41] Mihalis A. Nicolaou Student Member,Hatice Gunes, Maja Pantic. Continuous Prediction of Spontaneous Affect from Multiple Cues and Modalities in Valence?Arousal Space. IEEE TRANSACTIONS ON AFFECTIVE COMPUTING, VOL. , NO. , MONTH , 2011.

Appendices

A

Contents of enclosed CD-ROM

On the enclosed CD-ROM, the entire source code of the project is included. This consists of the Emotiboard project, the used version of openEAR. The Main source files of Emotiboard project are located in ”CD: Emoitboard” folder. The source folder includes AutoEmotiRecognition, chf and com packages. Each package also has its own sub directories and related java file. Here is the description of the contained java files:

Package autoEmotiRecognition:

AnimationPanel.java: panel to update animated avatar

EmoSpeechPanel.java: panel is used to update emotional speech data EmoSpeechRecognition.java: Instance of the class to run SMILExtract.exe

Package emoticalreaction:

EmotionReactionApplication.java: the startup and main function of the Emotiboard, which is also a graphical window containing configurations.

Emotion.java: this class is used to store the information about emotions. Each instance is the data source for one EmotIcon

EmotiLogFormatter.java: the class is used to log processed data

Package emoticonboard:

EmoticonBoard.java: this window shows the emotion of the other client seen in the back- ground through Skype

EmoticonBoardPanel.java: the panel to show the reaction of other client EdaChangesImgae.java: this class is used update drop images on the smilyface

Package qSensor:

QDataPanel.java: a Panel to visualize the data obtained by QSensor, inherits JPanel prop- erties

QDataParser.java: this class is used to parse the data received from qSensor

QSensorCommunication.java: this class open the communication with the Serial Port which is used to read the data from Bluetooth in order to read data from qSensor

TimeSeriesRenderer.java: A time series chart with a custom renderer that varies the shapes and fill colors by item

ServerApplication.java: constructor of the ServerApplication to run the server part

Package client:

App.java: the core class to connect all other classes and client with each other

AppPanel.java: the content pane of the application subclass this class to design the graphical part of your application

Client.java: it represents one machine at least one application instance running on a machine. This class is in charge of connecting and communicating to the server, but also to parse the messages received from the server

ConToServerPanel.java: creates a graphical interface to connect to the server

Package server:

ClientThread.java: thread object for each client connected to the server

Server.java: the server part of the application. Its a multithread server and specifies one thread for each client

ServerParser.java: this class is a parser object for the messages are sent to the server. These messages are logged into the file, processed, and then they are broadcasted to the other clients

Package utilities:

FileHandeling.java: this class is used to handle the tasks related to saving or deleting the audio files

B

Questionnaire