CONCLUSIONES Y RECOMENDACIONES

By definition, the primary objective of sonification is to aid in the perceptual detection of salient features in the data. Thus, as will be argued below, it is somewhat surprising that so few EEG sonification papers have offered any quantitative or listening assessment of their sonification output. It is clearly important to test the ability of a sonification to convey the “signal” in the data in a manner that the human listener can perceive. Indeed, this would appear to be a prerequisite for any scientific work on sonification. This seems particularly important with EEG, given the very high temporal resolution and noisy nature of the EEG signal, by comparison with some other targets for sonification, such as seismological or stock market data. But a failure to validate sonifications applies more widely than to the field of EEG alone. In a systematic review of mapping strategies for the sonification of physical quantities, Dubus (Dubus and Bresin, 2013) makes the same complaint and can find only one example where two sonification techniques have been compared side-by-side.

Presented below is a summary of 5 papers that did conduct a perceptual listening test of the sonification output.

3.6.1.

Aesthetic Assessment

Wu, Li and Yao (2013) sonified Alpha EEG from participants with their eyes closed and eyes open and attempted to make the output of the sonification more „musical‟ by using artistic beats and tonal filters.

Subsequently, 22 participants were played 4 different 60-second sonifications of single- or multi-channel EEG from two conditions: eyes closed and eyes open.

criteria; tempo, valence, arousal, rhythm, musicality and richness, (These terms were not defined by the authors in this paper). The authors concluded that:

“… the notes in eyes closed music were longer in duration, lower in pitch and slower in tempo, which demonstrated a peaceful and quiet mood corresponding to the eyes closed state. In contrast, the notes of eyes open were shorter in duration, higher in pitch and faster in tempo, which meant that the brain was relatively alert and active.”

But, not pointed out by the authors was that this musical correspondence is a product of the sonification mapping, not any intrinsic musical properties of the signals corresponding to eyes closed and eyes open brain activity, thus rendering the assessment of the sonification aesthetic qualities somewhat superfluous.

3.6.2.

Two-alternative Forced-Choice Method (2AFC)

In a paper by Loui (Loui et al., 2014), fifty-two naive participants were given a „two alternative forced-choice test‟, where they were asked to listen to several 10 second sound files of sonified EEG and for each one, to choose if the file contained epileptic seizure activity or not.

The experiment consisted of three separate blocks in one session. In the first block, without any training, participants listened to 26 sound files, half of which contained epileptic seizure activity and the participants had to choose if the file had seizure activity or not. In the second “Training” block, 3 sound files with and 3 without seizure activity were played and the participants were informed which category the files belong to. The third block was the same as the first but

after training. Loui showed that with a very short training protocol, participants were able to identify seizure activity at a better than chance level.

Vialatte et al. (Vialatte et al., 2009, 2012) took the EEG data from elderly patients suffering from mild cognitive impairment (MCI) who would go on to develop Alzheimer's disease within a year and a half and compared them to healthy age-matched controls. The 5-minute eyes-closed EEG data was reduced in complexity by a sparsification process called bump modelling that tries to highlight only the prominent features in the data set.

In a perception test after 30 minutes of training, Five listeners were played the sonifications of 5 MCI patients and 5 control subjects and asked to rate them as either “certainly MCI”, “unsure” or “certainly healthy”. Four out of five listeners classified all patients correctly, giving an overall error of 11%.

3.6.3.

Temporal Onset Detection

Khamis, Mohamed, Simpson, and McEwan (2012) sonified 2 channels of 24-hour EEG recordings from 17 patients with temporal lobe epilepsy by speeding the data up by 60 times to move it into the audible hearing range. This is called audification and is one of the simplest methods of converting time series data into sound. Khamis and colleagues then played the sound files to five listeners to see if they could detect the onset of the epileptic seizure activity and localise which hemisphere the seizure begins. After a 2-hour training session, the participants were played different examples of sonified EEG alpha, theta and delta waves, as well as movement artifacts and epileptic activity from 7 of the epilepsy patients. They then spent a mean of 17.2 hours listening to the remaining 10 epilepsy patients‟ EEG data. The listeners were able to detect the

seizure with a mean sensitivity (i.e., true positive rate) of 81.3% and a false positive rate of 0.012 per hour. The average lateralisation accuracy of epileptic seizure for all five listeners was 77.62%, with a standard deviation of 7.14%.

Khamis et al. went on to claim that:

“With a limited amount of training human listeners can identify seizures and seizure lateralisation from audified EEG signals from electrodes placed at P3-T5 and P4-T6 (left and right parietal and temporal lobes) with a sensitivity comparable to electroencephalographers /epileptologists detecting visually from EEG traces with 21 electrodes… with greater than a factor of ten improvements in the rate of false detections per hour”.

This is an interesting study in that it shows that inexperienced “listeners” can detect features in the EEG data from the simplest form of sonification, i.e., audification. Furthermore, these inexperienced “listeners” were able to achieve detection accuracies equivalent to trained EEG experts.

Alexis Kirke and Eduardo Miranda (Kirke and Miranda, 2012), attempted to sonify the emotional arousal and valence of Kirke while he was listening to ambient music, hard rock and silence. Arousal and valence can be inferred by the relative activity of the left and right frontal cortex with a metric called “frontal alpha asymmetry” (Davidson, 1998) see section 2.1.11 in chapter 2.

Three “listeners” were played two sonifications with five affective changes in each file. The task was to identify any perceived changes in valence and arousal of the sounds of the EEG data. Kirke and Miranda suggested: “there is an average of 80% communication rate for Valence and 70% communication

rate for Arousal.” Unfortunately they appeared to only count correct hits and not false positives and did not give much detail on the listening test.

3.6.4.

Key issues in the assessment of EEG sonification

In summary, to date some EEG sonification studies have used the „two- alternative forced-choice method‟ (2AFC) to assess a sonification‟s ability to convey information, e.g., distinguishing between patient with epilepsy versus a non-patient (Loui et al., 2014), or patients suffering from mild cognitive impairment versus healthy age-matched controls (Vialatte et al., 2009, 2012). Some of these studies have shown very good detection accuracy but this method does not really capture the temporal aspects of perception of the sonified data, an aspect of EEG sonification that we will consider below.

Some studies captured some of the temporal information by asking participants to identify the time of onset of a particular EEG activity. So, for example, Khamis (2012) played two channels of EEG sonification of patients with temporal lobe epilepsy and asked the study participants to push a button when they heard the onset of seizure activity. After only 2 hours of training, non-expert listeners could perform this complex detection task to an expert level. However, epileptic activity has significantly larger amplitude and a very different morphology compared to background EEG, and thus is easily distinguished. Although this is an important area for applying EEG sonification, it is also somewhat specialized, since epilepsy only affects around 1% of the population (Thurman et al., 2011).

From the point of view of assessing the temporal resolution of a sonification this kind of assessment has the potential to offer more information than the 2AFC

method, but it does little to assess the full range of dynamic characteristics of listening to continuous sound-based feedback.

Physiological data tends to be complex and noisy, consequently a person‟s response and their attempts to comprehend that data may be similarly complex. Unfortunately none of these assessment methods seems to capture the complexities or temporal dynamics of the listening task.

Thus the development of a methodology that could assess the ability of sonifications to convey temporally rich EEG data in real-time could greatly assist the design and selection of appropriate sonifications for a range of application areas such as neurofeedback, surgical monitoring, or brain computer interfaces (BCIs).

EEG Neurofeedback Sonification literature