In this part I expand on how the notion of metaphor – specifically what Lakoff and Johnson define as a conceptual metaphor (Lakoff & Johnson 1980: 8) – can be used to understand how we experience relations between different media. I begin with recent research into cross-modal perception – how the brain processes two or more different sensory modalities – and then borrow from Cook's idea of 'enabling similarity' (Cook 1998: 71), to try to understand the dynamic metaphor relationship between media and how it modulates through time.
On a neurological level, conceptual metaphors have been shown to have some kind of correlation to neural mappings in the brain (Feldman and Narayanan 2004:385). In 2001, neuroscientists Vilayanur S. Ramachandran and Edward Hubbard conducted an experiment (originally devised by Wolfgang Köhler in 1929) where they showed two shapes, one jagged and one rounded, to two groups (one American and one Tamil) and asked them "which of these shapes is 'bouba' and which is 'kiki'?". Both groups answered almost unanimously that the 'kiki' was the jagged one and 'bouba' the rounded. The researchers concluded that this shows 'synaesthesia-like mappings'
in the brain, where the visual and auditory cortices exhibit some kind of connectivity (Ramachandran and Hubbard 2001: 3). This was deemed to be evidence of the neurological basis of sound symbolism, an idea that vocal sounds are meaningful in themselves, that the brain creates cross-modal associations that are the basis for metaphor, and that synaesthesia and metaphor creation are linked.
More recently, neuroscientist Danko Nikolić has suggested that this experiment can be better explained by the concept of 'ideasthesia' rather than 'synaesthesia' (Nikolić: 2009). Nikolić defines ideasthesia as the phenomenon where concepts evoke perception-like experiences. Implicit in the idea of synaesthesia is the association of two sensory elements with little connection at the cognitive level; ideasthesia, on the other hand, puts emphasis on the cognitive aspect of cross-modal interaction, rather than the perceptual. Ideasthesia conceptualises the metaphorical connections of rational abstractions and the semantic links underpinning them. To clarify further, in the classic view our mind captures information, sounds, colours, smells, textures and tastes, and classifies them respectively as cicadas, blue, burning, rough or bitter. More recently, neuroscientists have concluded that these perceptions of the outside world are shaped by our conceptual understanding, that there is a constant feedback to our senses about what is being perceived, and that without the cognitive function there can be no perception. This implies a rich semantic network of both ideas formed by perception, and perception formed by ideas that underpin our understanding of metaphor, and suggests we do not need to be a synaesthete to have synaesthesia-like experiences, in other words, to have strong cross-modal metaphoric intuitions.
The importance of synaesthesia/ideasthesia as models of the experience of sound and image has been well documented and formed a basis of much multimedia work in the beginning of the twentieth century. In his book Analysing Musical Multimedia, Cook suggests that metaphor is a way of understanding how music and image interact, how we take ideas or emotions from one domain and map them onto another. Cook builds on the 'congruence-associationist' model of psychologists Sandra Marshall and Annabel Cohen (1988), which describes how meaning is ascribed to a film through music. In this model, attention is directed to the areas where music and film overlap, thus referential meaning associated with the music is ascribed to the overlapped audio-visual components. What is important here is that there is unidirectional transfer of meaning from one medium to the other, where there is perceived overlap of attributes. In Cohen's later study, "Congruence- associationist framework for understanding film-music communication" (2001: 259) she broadens the idea to a multi-stage model, involving 'bottom-up processing' (features derived from perceptions), 'cross-modal congruence' involving both semantic and syntactic features, 'top-down processing' (experiences in long term memory), and its interaction with consciousness and short-term memory.
Cook takes the idea of the 'congruence-associationist' model and proposes that it operates in a similar way to metaphor. He coins the term 'enabling similarity' to describe the way in which attributes common to two domains – the parts where they overlap – provide a basis for transfer of many other attributes not necessarily held in common:
The meaning of metaphor does not lie in the enabling similarity: it lies in what the similarity enables, which is to say, the transfer of attributes from one term of the metaphor to the other'. (Cook 1998: 71)
Once a link is created between concept A and concept B, through some idea of shared attributes, then there is the possibility of all of B's attributes being applied to A. Without oversimplifying what is a hugely complicated cognitive process, I would like to suggest that this unidirectional transfer of meaning, is one way we can understand what might be happening under the hood of consciousness when we are exposed to information on multiple sensory and cognitive levels. The fact that it is unidirectional does not undermine the idea that we might be experiencing this in terms of a constantly updated feedback loop of cognition, but it does underline the principle that a source and a target in metaphorical terms are never equal. For example, in the metaphor 'Time is Money', attributes of money are mapped onto our understanding of time, not the other way around: we do not see money in temporal terms (in this case), we see time in financial terms (Lakoff & Johnson 1980: 8).
In film music, is the music the source or the target? Are attributes of music used to understand the image or vice-versa? One would have to say that in the classic film- music relationship, where the narrative has the predominant role, the image is the target and the music is the source. Emotional attributes of the music are used to further understand the image. In a hypothetical scene where a young boy is mourning the death of his pet dog, the attributes of the music with which we associate the emotions of the boy give us an insight into his feelings of loss. We understand the meaning of the situation through the music. What is not the case, at least initially, is that we come to an understanding of the meaning of the music through the situation of the boy's mourning. This may be because the meaning of the music is not seen as paramount to the narrative; it is not our primary concern. The plight of the dog and the boy are.
Does this mean music will always adopt the role of 'source' to the 'target' of image or text? Not necessarily, because these kind of metaphoric relationships are dynamic and are constantly being updated. However, I would argue that they are never equal or balanced. Furthermore, they are often highly subjective, in that a viewer at any one moment, might be looking for one meaning through the image and another meaning through the music.
There is always a hierarchy in how source and target (in the terminology of metaphor) are defined.34 In language, this is created by the syntax: the order of the words. But how does this manifest in a non-linguistic idea of metaphor?
The hierarchy of relationships between different media depends on many factors. Primarily, the entrenched historical form of a given genre is what mostly determines our point of view. These conventions play a large part in determining our position in the experience of multimedia. In much new audio-visual art the relationships between the media are not as entrenched as, for example in mainstream cinema, theatre, or even mainstream music video, so there can exist a greater dynamic range in how the hierarchies are constructed.
In the music-text-filmpiecescited in this paper, in which at least three distinct media interact in our minds, the metaphoric hierarchy of these relationships – what is the source and target – play an important role in how the meaning of the work is eventually constructed. This is specifically what creates 'focus' in the work, affecting the perspective of the spectator. We might understand the music through an idea in the text, or the text through an idea in the music, or the music through a visual idea etc. Furthermore, these relationships are always in a state of flux. I have observed several ways in which these metaphorical hierarchies are constructed and modulated in the context of my music-text-film pieces, which can be applied to audio-visual forms in general:
1. Context: As suggested in the subsection 'Melos Lexis Opsis', context plays an