The first two chapters, following this introduction, will review experimental and computational studies on music and emotion. Chapter 2 presents the theoretical and experimental background for this thesis. First, human emotions are discussed at the neurobiological level. Then, a description of the main mechanisms of emotion serves as an evaluation of the different mechanisms by which music can convey emotion to the listener, and how they can be quantified. This chapter includes an overview of the most common methodologies used to quantify musical emotions, with a special emphasis on methods based on the subjective feelings of the listener and upon physiological activation.
It will be shown that the complexity of experimental data on music and emotion studies, require capable methods of analysis, which allow for the extraction of relevant information from experimental data. In Chapter 3 a class of spatiotemporal (Kremer, 2001) connectionist models will be proposed as a capable paradigm to analyse the interaction between sound features and the dynamics of emotional ratings. The proposal of using computational models for this study aims to investigate the relationships between music structure and
emotion in more detail and includes their computational (abstract) representation. In the following three chapters I will present new computational and experimental studies. Chapter 4 describes a new neural network model of musical emotions, which accounts for the subjective feeling component of emotions. The experimental data for this simulation experiment was obtained from a study conducted by Korhonen (2004a)1. The neural network is trained to predict affective responses to music, based on a set of psychoacoustic components extracted from the music stimuli. An analysis of the network dynamics provides new information about the relationships between the sound and its affective dimensions.
In Chapter 5 a new experimental study is presented which is based on the continuous response methodology (Schubert, 1999b) to obtain a listeners affective experience with music. Participants were asked to report their subjective feelings while listening to music, while at the same time, their heart rate and skin conductance levels were recorded. Evidence provided in this study suggests the existence of relevant interactions between the psychological and physiological components of emotion.
This data is used in Chapter 6 to develop a new computational investigation and for the extension of the neural network model presented in Chapter 4, to include physiological cues. The aim of this chapter is to verify if physiological activity has meaningful spatiotemporal relationships with the affective response.
The last chapter of this dissertation, Chapter 7, summarises the research, and discusses its implications and contributions to the field of music perception, emotion and cognition. A set of recommendations is also included which discuses some of the potential extensions and applications of this model.
The nature of emotion in music:
Theoretical and experimental
investigations
As a background to this thesis, this chapter presents a literature review of the state of the art research and ideas on musical emotions. Due to the multimodal nature of emotions, modern research is focusing on particular components of its modalities and experiential counterparts (such as feeling states or other manifestations). Initially, the neurobiological framework for emotions theorised by Damasio (1994, 2000) will be used as the substrate for a conceptualisation of emotions, their multimodal nature and classes of inducers. As it will be discussed, emotions arise as a construct of complex psychological and physiological processes. An overview of these mechanisms is given in this chapter, as well as an examination of their potential involvement in musical emotions, and the focus on auditory stimuli as potential emotion-competent-stimuli (Damasio, 2000). The discussion on whether music induces emotions, or merely represents them, will also be discussed in the light of “emotivist” and “cognitivist” perspectives on musical emotions (Kivy, 1989). The last part of this chapter, discusses the main
methods to measure emotions experimentally in music research.
2.1
A substrate for emotions
In the very influential book “The Feeling of What Happens: Body, Emotion and the Making of Consciousness”, Damasio (2000) proposed a neurobiological framework in which he demonstrates that certain organisational principles in the brain might reflect emotional states. He attributes to emotions a major role in the general economy of the mind, by suggesting that emotions and feelings are part of the neural machinery for biological regulation (whose core is formed by homeostatic controls, drives and instincts). Emotions are also inseparable from the idea of reward or punishment, of pleasure or pain, of approach or withdrawal, of personal advantage or disadvantage.
Within this framework, emotions are complicated collections of organised chemical and neural responses with some regulatory role to play, leading in one way or another to the creation of circumstances advantageous to the organism. The biological functionality of emotions include the production of a specific reaction to the inducing situation (e.g. run away in the presence of danger), and the regulation of the internal state of the organism such that it can be prepared for the specific reaction (e.g. increased blood flow to the arteries in the legs so that muscles receive extra oxygen and glucose, in order to escape faster). Damasio (2000, pp. 68-69) summarises the sequence of events in the process of emotion in the following manner:
1. “engagement of the organism by an inducer of emotion” (e.g. a particular object processed visually);
2. “the signals consequent to the processing of the object’s image activate all the neural sites that are prepared to respond to the particular class
of inducer to which the object belongs”. These sites have been “preset innately”, although past experience has modulated the manner in which they are likely to respond;
3. “emotion induction sites trigger a number of signals toward other sites (for instance, monoamine nuclei, somato-sensory cortices, cortices) and toward the body (for instance, viscera, glands)”.
Damasio categorises emotions into three categories related to their “biological functionality”: background, primary (or basic) and secondary (or social). Background emotions are defined as certain responsive conditions of the internal state engendered by ongoing physiological processes, or by the organisms’ interactions with the environment, or both. These emotions endow us with, among others, the background feelings of tension or relaxation, of fatigue or energy, of wellbeing or malaise, of anticipation or dread. They target more internal rather than external processes. They are also richly expressed in musculoskeletal changes, for instance through subtle body postures and overall shaping of body movement. Primary emotions (fear, anger, disgust, surprise, sadness, and happiness) are considered to be what other studies call the basic emotions, a set of shared predispositions that allows us to respond in a more or less stereotyped way when certain features of stimuli are perceived. The limbic system is often associated with this process, as a unit involved with the detection of these features, and the source of certain brain signals that alter current cognitive activity and activate certain biological responses.
On the top of these two groups appear the secondary or social emotions (sympathy, embarrassment, shame, guilt, pride, jealousy, envy, gratitude, admiration, indignation, and contempt). Rather than predispositions these constitute “learned” connections related to the state of the organism and the stimuli that triggered the emotional process, both at conscious and unconscious levels. From this category of emotion Damasio derives his “somatic markers
hypothesis” (Damasio, 1994), a refinement of the mind to assess and respond quickly to a set of conditions of the internal and external environments of an organism, that are associated with the detection of an “emotional-competent- stimulus”. In situations that require decisions involving complex and conflicting alternatives, cognitive processes may become overloaded, and the somatic markers can aid the decision process. Such associations are reinstated physiologically and drive cognitive processing when needed towards the selection of the appropriate action.