TAQUILLES I SISTEMES .1 SITUACIÓ ACTUAL

The EyeMusic sonifies a 24x40 pixel coloured image, typically over 2 seconds (Levy-Tzedek et al., 2012a, 2012b). Colours are first segregated into one of five categories (red, green, blue, yellow & white) and each category is encoded through timbre (e.g. white = piano, blue = marimba). The colour’s vertical location is denoted through the pitch / note of the instrument, using notes across eight octaves of a pentatonic scale (higher pitches reflect higher spatial positions) while the brightness of each colour affects the loudness of the note (bright is loud and dark is quiet). Finally, each column is sequentially presented over time to complete the X-axis.

Initial experiments with the EyeMusic have focused on sensorimotor interactions between behaviours informed through either vision or the EyeMusic. Levy-Tzedek et al. (2012a) report a comparison between visually-informed reaching behaviour and SSD-informed reaching behaviour for a single white target object. Feedback for incorrect final hand positions was either on screen for the

visual condition or sonified through a blue object for the EyeMusic condition. They found no significant difference between vision and the EyeMusic trials in users’ movement time, peak speed and path length, indicating that EyeMusic’s encoding could inform users’ motor movements. However, there were significantly higher errors on endpoint localisation than with vision. A subsequent project by Levy-Tzedek et al. (2012b) examined the shared spatial representation of vision and EyeMusic soundscapes. Participants were required to indicate the location of an abstract target (either visually shown or heard using the EyeMusic) using a joystick. They found that altering the visual sensorimotor feedback (in this case, skewing the joystick’s representation on screen by 30°) influenced not only the visual trial movements but also the EyeMusic trials (where no feedback was given). The former experiment indicates that soundscape feedback utilising timing, timbre, loudness and pitch can inform a spatial representation for movement, while the latter experiment indicates a shared spatial representation between vision and the EyeMusic soundscapes, as influenced by external feedback. While both of these experiments use simple target shapes on black (silent) backgrounds, limited colours and spatial target locations, these experiments demonstrate sensorimotor guidance that may be applicable to more natural complex environments where colour is meaningful rather than arbitrary.

Table 2.2. Coding strategies in current sound-colour SSDs.

SSD Spatial Representation Colour

Representation

Audio-Colour Coding Reference

SoundView A single colour is selected via a stylus applied to a 2D tablet.

Hue, saturation and brightness.

White noise first goes through a low-pass filtering cutoff frequency denoting brightness1. Hue values determine which of 12 pitch-filters is applied5 and this effect is modulated by saturation1,4.

Doel, 2003.

Eyeborg A single colour is made from a centrally-weighted distribution of pixels.

Hue and saturation. Hue is either categorised into 12 or 360 microtones (1st version) or mapped to a continuous frequency between 363.797Hz and 717.591Hz (2nd version)5. Saturation is denoted through volume4.

Else, 2012.

Kromophone Single point. 2nd version codes the horizontal axis over time.

1st version uses hue, saturation and luminance. 2nd version uses red, green, blue, yellow and luminance.

1st version maps hue to pitch5, saturation to left-right panning and luminance to volume2.

2nd version maps the contribution of red, green, blue, yellow and luminance to the volume of five unique sounds (each with unique panning, pitch and timbre6).

Capalbo & Glenney, 2009.

See ColOr Horizontal axis coded via interaural timing differences. Depth coded through tempo. Global modules can select features for sonification.

Hue is transformed into 7 focal colours. In addition saturation and luminance are used.

The seven focal colours are given a unique timbre6 each with their volume increasing nearer the colour’s focal point. Increased saturation is communicated through higher pitched notes3. Finally darker colours add a double bass while brighter colours add a singing voice, with brighter and darker variants using higher and lower pitched notes respectively1.

Bologna et al., 2007.

ColEnViSon Horizontal axis over time. Vertical axis over audio channels. Bottom-up orientation modules select salient regions.

CIE LCH space is used (luminance, chroma and hue) to derive 10 focal colours.

Each colour is categorised according to its proximity to 'colour centroids' of focal colours. From this, unique timbres are given to 10 colours including greyscale6.

Ancuti et al., 2009

EyeMusic Horizontal axis over time. Vertical axis coded via note on 8 octave pentatonic scale (pitch).

Red, green, blue, yellow and white categories as well as a brightness dimension.

Each colour category is given a unique timbre6 with the brightness of the colour denoted through its volume2. A colour's vertical position denotes its pitch on 8 octaves of a pentatonic scale7. A single vertical column is sonified with subsequent columns presented in a serial fashion over time.

Levy-Tzedek et al., 2012a, 2012b.

2.4.2. Potential future directions

Whereas many of the future directions for colour-to-touch depend on technological developments, the same does not apply to colour-to-sound as complex sounds are relatively simple to produce. Instead, it is suggested that future developments in colour-to-sound translation will primarily be based on finding optimal solutions with respect to human perceptual abilities.

The idea that there could be a law of association between colours and sounds (physical or psychological) has a very long history (Jewanski, 2010). In multi-sensory perceptual interactions, there are automatic associations between higher vertical locations with higher pitches (Melara & O’Brien, 1987; Roffler & Butler, 1967; Walker et al., 2010), as well as brighter luminances with higher pitch (Hubbard, 1996, Marks, 1974) and increased loudness (Lewkowicz & Turkewitz, 1980, Marks, 1987). These associations are exploited in many of the SSD devices described here. When asked to match colours with sounds, there is also a tendency to link saturation with pitch albeit non-linearly (‘middle C’ pitch is linked to the highest saturation, higher or lower pitches reduce saturation) and to timbre (pure tones are less saturated than instruments) (Ward, Huckstep & Tsakanikos, 2006). As for hue, one possibility is that this could be linked to various aspects of timbre. Timbre is formally defined as that quality of sound that enables different instruments to be distinguished (irrespective of loudness or pitch).

A higher-level approach to timbre is to get participants (music students, in this example) to rate sounds along different descriptive labels (Zacharakis, Pastiadis, Papadelis & Reiss, 2011). Using factor analysis they found three factors explaining 82% of the variance in timbre-descriptive terms: namely, volume / wealth (descriptions of ‘full’ and ‘rich’), brightness / density (‘brilliant,’ ‘deep,’ ‘bright’ and ‘thick’) and texture / temperature (‘soft,’ ‘harsh,’ ‘rounded,’ ‘warm’ and ‘sharp’). These orthogonal dimensions could potentially be mapped to red-green and blue-yellow dimensions to create a circular representation of hue.

In utilising perceptual aspects of timbre for the representation of colour, the most salient dimensions for human perception should be considered. In practice, timbre depends on the distribution of frequencies and how these change over time to create a spectral envelope. The spectral envelope for any given instrument timbre contains the ‘attack’ (change from silence to peak volume),’ ‘decay’ (reduction of initial volume after peak), ‘sustainment’ (steady sound levels from holding a note) and ‘release’ (final reduction of sound levels to silence). Systematically varying these components offer new possibilities for the representation of colour.

Considering the distribution of frequencies in timbre, Von Bismarck (1974) identified four dimensions that explained 90% of the perceptual variance in artificially created steady-state timbres.

The two most influential dimensions were ‘sharpness-dullness’ (centre of gravity of the spectral distribution, dominant upper or lower harmonics), and ‘compactness-scattered’ (compact refers to harmonic combinations of frequencies while scattered refers to a wide selection of frequencies to create white noise). Since either end of these dimensions provide perceptual opposites (e.g. dominant upper harmonics is opposite to dominant lower harmonics) the use of these opposites can be used to represent opponent colours present in veridical colour perception such as yellow and blue. In addition, if the other dimension is positioned as a perpendicular axis, this provides another ability to represent opponent colours (e.g. red and green) through perceptually opposite sounds (harmonics and noise). In combination this soundspace provides neighbouring sounds that are more similar than their opponencies, which mimics the structure and position of opponent colours in veridical colour perception.

Considering the temporal dynamics of timbre (attack-decay-sustainment-release profile), Wessel (1979) found, using multi-dimensional scaling, that the temporal dimension was mostly defined by instrument family, noting three distinct groups (Trumpet-trombone-French horn; oboe- bassoon-clarinet; violin-viola-cello). Grey (1975, 1977) found that three dimensions could explain the majority of variation in dynamic timbre (with equalized volume, pitch and duration), these being the overall spectral energy distribution and, over time, the synchronicity of upper harmonics (where high frequency sounds all increase and decrease in volume together) and low energy high frequency noise during the attack segment (subtle disharmonic content present just as the note is played). Barrass (1997) used Grey’s timbre dimensions to create 3D space containing instrument sounds at specific locations so that similar sounds would be spatially close together and dissimilar sounds would be far apart. The purpose of this was to mimic the structure of HSL colour space but using sounds instead of colours. He matched luminance-pitch (linear), saturation-‘upper/lower overtone emphasis’ (linear) and hue-timbre (circular) dimensions to create a perceptually distinctive information sound space. In order to construct a circular representation of hue from timbre, two opponent axes (similar to RG and BY axes in CIELab space) were created from ‘upper harmonic synchronicity’ and ‘high frequency noise during the attack phase’ to select the appropriate instruments (Grey, 1975). As a result, the relationship between two colours (e.g. are they perceptually close or far apart?) could be understood through how the sounds from those colour locations relate to one another in terms of perceptual similarity or dissimilarity.

The ability to discriminate two sounds from one another is an essential characteristic in determining that these sounds could represent different properties, such as two colours that differ from one another. But while the sounds might indicate that ‘sound A’ represents something

different than ‘sound B,’ it does not by itself indicate what specific shades of colour sounds A and B might represent. One way that you can imply that certain sounds represent certain colours is through using pre-existing intuitive associations between sounds and colours in your cross-modal mapping. One way of establishing any cross sensory is simply through asking large groups of participants which colours they think would ‘best suit’ a variety of timbres. Rossi, Perales, Varona and Roca (2009) explored the colour associations found for instruments (played in a melody). They note that some associations occur more often than expected by chance (e.g. guitar=red, vibraphone=yellow). Currently, there is no indication as to why these associations would be this way around, so further investigations should seek to understand the rationale behind these choices made by participants and whether or not the use of these affects individuals’ ability to learn colours through sound.