Images have for the most part been studied as representations rather than as interac- tions. The key questions of image analysis have been: ‘What people, places and things are represented in this image?’ and ‘What kind of connotative or symbolic meanings are associated with these representations?’ (Van Leeuwen and Jewitt, 2001: 92). But clearly images are also used to do things to or for or with people: to persuade (adver- tisements), instruct (patterns for dress making), explain (diagrams in textbooks), warn (image of skull and crossbones on a door), and so on. Kress and Van Leeuwen (1996: 121) have extended Halliday’s theory of ‘speech functions’ to images. Like linguistic representations, images can also either ‘offer’ or ‘demand’. But where, in the English language, this is realized linguistically, in images it is realized by the system of the gaze. If a person (or animal) represented in an image looks at the viewer, the image realizes a ‘demand’: ‘the gaze … demands something from the viewer, demands that the viewer enter into some kind of imaginary relation with him or her’ (ibid.: 122). If such a look is not present, the image is an ‘offer’. It then ‘“offers” the represented participants to the viewer as items of information, objects of contempla- tion, impersonally, as though they were specimens in a display case’ (ibid.: 124).
The precise nature of such offers and demands comes about through a combination of different visual and contextual features, just as in the case of the speech act. The picture in figure 6.1, for instance, combines a visual demand with a pointing finger, an imperious look, and a military moustache and uniform – both symbols of authority. Together these features modulate the demand into a very direct, maximally authorita- tive visual summons.
The linguistic speech act in figure 6.1 expresses another kind of demand. It is a state- ment with direct address, a verb which realizes a kind of request – ‘need’ – and an agent – ‘your country’ – who has, in the given context, the right to command the addressee – a moral right, based on patriotism. Taken together, these features create a hybrid speech act, a speech act which oscillates between bluntness and formality, directness (the direct address) and indirectness (the indirect demand). And then I have not even mentioned the typography, with its highly salient, large ‘you’.
The question is, what do we have here? One or two speech acts? The same demand formulated twice, once in a rather direct way, and once verbally, in a more indirect, less personalized and more formal way? Or one single multimodal communicative act in which image and text blend like instruments in an orchestra?
If we take the first approach we will have to sequentialize what we in fact see in one glance and posit some kind of reading order that leads the eye from the picture to the text. The text then becomes a kind of caption for the picture. But is this the right approach? In the everyday face-to-face equivalent of the poster we would not dream of doing so. Imagine an actual uniformed man addressing us in this way. Clearly we would experience it as a single, multilayered, multimodal communicative act, whose illocutionary force comes about through the fusion of all the component semiotic modes: dress, grooming, facial expression, gaze, gesture – as show in figure 6.2. Perhaps the same approach should be used here. Perhaps speech acts should be renamed communicative acts and understood as multimodal micro events in which all the signs present combine to determine their communicative intent.
The example shows how many linguistic and visual features can combine into a single communicative act, and how many different kinds of communicative act can therefore be expressed. It also is an example which may at first sight seem monologic
image:
finger pointed at the viewer unsmiling look at the viewer military uniform
military moustache
language:
2nd and 3rd person combines declarative
lexicalized request ( need ) authorized subject ( country )
typography:
large and heavy you
stylistically cohere into single communicative act
rather than dialogic. Yet it does of course have a preferred response – enlist in the Army – and hence also a dispreferred one.
So far I have mentioned speech acts, image acts and acts of non-verbal communi- cation, such as facial expressions and gestures. But does the idea of the communica- tive act extend to all semiotic modes? Can we have ‘music acts’, for example? Or ‘sound acts’? Or ‘smell acts’? In Van Leeuwen (1999: 92) I began to look at the role of melody in realizing ‘music acts’. Melodies often have an overall direction. They can tend upwards or downwards. In singing, going up in pitch requires an increase in vocal effort, and going down a decrease, a degree of relaxation. As a result rising melodies can be used to energize people, to rally people together for a common cause, for instance in national anthems and patriotic songs, or in upbeat advertising songs. Melodies with a descending pitch, on the other hand, can relax and soothe, and may therefore be used, for instance, in meditative, inward-looking hymns or in lullabies. The same applies to voices (see Van Leeuwen, 1992). The disc-jockeys of ‘Beautiful Music’ radio stations not only speak in low, relaxed voices but also allow their voice to go down a lot, so as to soothe and relax the listener, to ‘sound like a warm, friendly adult’ and to portray a ‘relaxed manner’ (ibid.: 237) while the disc-jockeys of Top-40 breakfast programmes speak at a pitch level well above their normal register and make much use of rising pitch, in order to energize the listener, to sound ‘up’ – ‘Your sound has gotta be a “hey let’s get it happening” sort of approach’ (ibid.: 238). Such melodic features combine of course with features of tempo, rhythm, voice quality, etc. Like speech acts and image acts, ‘sound acts’ result from a combination of features.
Finally, as far as smell is concerned, we live in a world where supermarkets pump chocolate flavours and other enticing smells into their air-conditioning systems to encourage customers to buy more, where one debt collecting company was reported to have impregnated the paper of final notices with a synthetic version of human sweat to instil fear, and where perfumes help us express our identities and attract the other – or the same – sex. Smell does not just ‘represent’, or ‘remind’ us of other places and times, it is also and quite deliberately used to do things to, for or with people.
To summarize: