Informaci´on imperfecta

Following past research, it seems possible that emotion processing happens in a hierarchical manner: from sensory processing to broad emotion discrimination finishing with narrow emotion- category distinguishing. For voices, this has previously been summarised by Schirmer and Kotz (2006), who have created a working model of temporal emotion processing stages for prosody based on previous research (for illustration purposes see Figure 5.1 from Schirmer & Kotz, 2006).

170 According to Schirmer and Kotz’s (2006) three-stage model, the first voice processing stage peaks at around 100ms and defines the analysis of acoustical features such as sound intensity during initial sensory processing of voices as reported by Engelien, Schulz, Ross, Arolt and Pantev (2000). This stage would not be modulated by emotional content. Next, early emotion effects visible during the discrimination of emotional (e.g. angry) versus neutral prosody reflect emotion significance evaluation within the superior temporal sulcus (STS, Grandjean et al., 2005).The final stage then describes discrimination between individual emotion categories such as anger and happiness at around 400ms which includes higher cognition and conscious semantic processing of prosody in more frontal areas. For example, the inferior frontal gyrus (IFG) in the frontal lobe has previously been found to be active during the processing of semantic information (Schirmer, Zysset, Kotz, & von Cramon, 2004).

In support, Iredale et al. (2013) have recently confirmed Schirmer and Kotz’s (2006) hierarchical emotion processing model in voices. Their findings from an ERP study suggested early emotional versus neutral discrimination of prosody in parietal regions, followed by more specific discrimination of happy and angry prosody and an activation shift to more frontal brain regions during the final stage of semantic processing. However, in contrast to Schirmer and Kotz (2006) – Iredale et al. (2013) reported emotion versus neutral prosody discrimination that was already visible during the first processing stage at around 100ms after stimulus onset. Note, however, that the auditory stimuli used in the current study are not based on semantic or verbal content.

Overall then, it seems likely that for voices, there may be three distinct stages of emotion processing: initial sensory processing which may are may not communicate emotional intent, a subsequent stage specific for the perception of emotion and a final stage of emotional evaluation that allows the differentiation between different emotion categories. Do those three primary stages of emotion processing also apply to emotions processed in non-verbal affect bursts and in faces? Some of the past research certainly suggests hierarchical processing of emotion across modalities: For example, Batty and Taylor (2003) reported earliest emotion effects in face expressions within the first 100ms, followed by specific emotion discrimination after 140ms after stimulus onset and a third stage which was defined as late frontal activity after 330ms. This late stage seemed to be biased towards processing negative emotions expressed in faces. Assuming an emotion-general mechanism across modalities supports the idea of a hierarchical three-stage model of emotion processing that may be independent of modality. However, to the best of the researcher’s knowledge, this claim of

hierarchical emotion processing across several modalities has not previously been tested in a within- subject design.

Indeed, no one has previously investigated in a within-subject ERP design whether emotional stimuli derived from different sensory modalities show comparable temporal patterns of processing. Very recently, Bayer and Schacht (2014) attempted to investigate underlying event-related potentials in response to rating emotional faces, pictures and written words.

171 To the best of the researcher’s knowledge, this is one of the first accounts to compare three different types of emotion stimuli in a within-subject design during an ERP study – although all within the visual modality. Promisingly, results suggested that all stimulus types showed comparable emotion- typical activity in early posterior negativity component (EPN) as well as in the late positive

component (LPC). However, whilst words showed superior processing for happy stimuli as evident in larger amplitudes for positive words, there were larger amplitudes for angry faces and pictures

eliciting anger which suggests stimulus-specific processing differences at a lower processing level. Further, by distinguishing intact faces and words from distorted faces or pseudowords (Schacht & Sommer, 2009), increased posterior negativity at occipito-temporal sites was evident for positive versus negative and neutral words between 388 and 438ms as well as for positive versus negative and neutral faces between 128 and 172ms.

Hence, those previous results are promising as they suggest that different stimulus types within the visual modality have comparable stages of emotional processing; however, depending on the nature of stimulus, this may occur at different latencies. Consequently, although auditory and visual stimuli differ significantly in their basic perceptual features, it is of interest whether emotion signals within voices and faces may be processed on a supra-modal level with comparable emotion- processing stages.

5.1.4. The Present Study

As evident from past research, several processing similarities between emotions presented in faces and in voices have been established. For example, not much emotion activity seems to be noted within the first 100ms after stimulus onset in either modality. Secondly, both the face and the voice- sensitive stages N170 and FTVP may – or may not – already be modulated by emotional content of the stimulus. Lastly, both modalities have been shown to process emotions in a hierarchical manner, starting with broad discriminations between emotional and neutral stimuli and narrowing down to individual emotion category distinction at around300ms after stimulus inset. This may involve a shift from sensory and perceptual to more cognitive emotion processing.

The main research aim of this present study is to investigate in a within-subject design whether the pattern of neural activity is comparable across two separate modalities. To keep variance between modalities limited, all stimuli were based on non-verbal emotion communication. In line with behavioural findings from the previous chapters in this thesis, shared cognitive mechanisms should result in comparable emotion processing patterns across early and late time-windows, independent of the modality of presentation.

172 Further, according to Schirmer and Kotz’s (2006) model of hierarchical emotion processing in voices, it is of interest whether both modalities show a hierarchical processing of emotion information, starting with broad processing such as emotional versus neutral discrimination and proceeding to more narrow emotion processing such as the discrimination between happy and angry stimuli.

In order to determine the onset of emotion processing abilities in faces as well as in voices, three time-windows have been determined for the present investigation. The first stage included a pre- emotion period, ranging from 70 to 130ms which is thought to reflect sensory processing in the P1 component (Luck, 2005). Further, in order to investigate whether the voice- as well as face-specific components N170 and FTVP are moderated by emotional content, the second stage was defined between 130 – 200ms. Lastly, in order to investigate the onset of emotion-specific discrimination such as between happy and angry stimuli, the late processing stage was defined between 300 and 600ms post stimulus onset.

The current within-subject study predicts the following:

Hypothesis 1: In line with supra-modal emotion networks (see behavioural results from previous chapters), emotional faces as well as emotional non-verbal vocalisations will show comparable temporal stages of emotion processing across all three time windows.

Hypothesis 2: According to the hierarchical model of prosody processing (Schirmer & Kotz, 2006), earliest emotion processing will not occur within the first 100ms after stimulus onset and then begin with broad discrimination between emotional and neutral stimuli. Later cognitive stages after 300ms after stimulus onset will then show narrower emotion discrimination between angry and happy stimuli. Again, this pattern will be visible within both modalities.

173

5.2. EEG Methods & Task