In this section I discuss the observations I made through experimenting and performing with both touch screen and MIDI controller performance interfaces throughout the performance iteration cycle discussed in Chapter 5.
7.3.1 MIDI Controllers
As a result of my background as a jazz instrumentalist, the tactile response of a MIDI controller facilitated an intuitive and expressive performance interface. Careful consideration went into creating the ideal mappings between control elements and specific functionality in Kortex. Once this stage was complete I was able to practise forming a relationship with the interface as if it were a traditional instrument. In doing so, I discovered which combinations of controls enabled the most expressive control over specific types of functionality in Kortex. For example, although both pots and sliders provide subtle and fine manipulations over parameters, sliders can be interacted with in a more aggressive fashion. Likewise, while the tactility of pads facilitates an ideal medium to rhythmically trigger banks of presets, they lack the visual feedback needed for triggering audiovisual content.
A delicate approach is required to maintain a balance between immediacy, expressibility and control as the number of parameters increases in the software instrument. A program like Kortex has upwards of one hundred controllable parameters that need to find space on the performance interface. PadModes was developed to circumvent this problem, but there is a limit to how many alternate states a set of controls can have before functionalities are buried so deep that immediacy suffers. Furthermore, the complexity of accessing these states limits the real-time creative potential of the instrument. Nevertheless, I believe the MIDI mappings outlined in section 6.4.3 balance these factors successfully and provide the most immediate, expressive and appropriate mappings between Kortex’s functionality and the interface.
7.3.2 Touch Screens
Initially in the first iteration I made exclusive use of a Jazz Mutant touch screen for my performance interface. As previously mentioned, fine parameter control over variables required that I diverted my attention from engaging with the visuals to correctly position my finger over the desired slider on the interface. Consequently I was not able to interact and engage simultaneously with the content I was producing, leading me to abandon touch screens as a performance interface.
Up until recently Kortex has been performed on single-screen setups. This meant I could display visual content via the HDMI output while using the laptop screen to display the grain reordering patterns and audiovisual thumbnails on the GUI. However, the triple-screen performance in Sydney for iteration 4 meant all the available outputs were used to send visuals to the projectors. This removed the ability to utilise a GUI to provide visual feedback and impacted my creativity during the performance. As a result I reconsidered utilising a touch screen interface to facilitate the required visual feedback. In doing so, I have found triggering audiovisual content and manipulating grain reordering patterns to be much more expressive and immediate than using a mouse to interact with an onscreen GUI. For example, I am able to easily select and punch in new audiovisual content in a rhythmical manner enabling me to play the interface as if it were a drum machine. Furthermore the interface seen below in Figure 73 provides instant visual feedback regarding the next 60 frames of audiovisual grain rearrangements. The visual feedback provides a highly intuitive approach for rearranging grains and encourages play and exploration.
Figure 73: Touch screen interface for manipulating the grain reordering patterns
The need to set up a local area network for communicating with the computer is the one downside to using a wireless touch screen device. Although most of the time this is not an issue, it does add extra stress when only having a few minutes to set up before performing. This can be alleviated slightly by using an ethernet cable, yet still requires configuration of the TCP/IP port, address and subnet mask for the OSC messages to be correctly received by the computer. I use an ethernet cable to connect the touch screen interface to my computer for two reasons. Firstly, it enables a more secure and reliable method to transfer messages, especially when playing in venues where nearly every audience member has a smartphone connected to a wireless network. Secondly, the latency of sending OSC messages over an ethernet connection is as low as 2 ms (Schmeder et al. 2010), so touch screen gestures and their associated control are perceived as immediate by the performer.
7.4 Audiovisual Relationships
The essence of my research was my investigation of how to create connections between image and sound so that the resulting construct could be perceived as a whole. During this process I discovered three different approaches that facilitate the formation of connections between the two media. These were defined by Edmonds et al. (2005) as being mathematical, metaphorical/intuitive or intrinsic. A mathematical relationship exists when the connection between audio and visual parameters is
expressed by an equation. Metaphorical and intuitive relationships are commonly found in film and are what I am referring to when describing perceptual characteristics. Finally, a relationship is said to be intrinsic when audio and visual media are synthesised from the same source.
Obviously genuine integration is very complex and nuanced and can be approached from many angles. I chose to take the metaphorical and intuitive approach to exploring relationships between image and sound. However, these relationships are bound by mathematical mappings that enable them to be perceived as emerging from the same source. Regardless of the approach taken, it is crucial that the resulting correlation can be perceived by an audience as an integrated audiovisual whole. Although the two media can be experienced separately from one another, the correlation seeks to strengthen and reinforce the emotional and aesthetic perception of the sensory object.
I have found modular routing interfaces facilitate an ideal approach for exploring various correlations between sound and image characteristics. The audiovisual routing interface described in section 6.2.2, and the JenOSC routing interface described in section 6.3.5, have become indispensable tools for instantly forging connections between the two media. I would recommend other audiovisual artists adopt a similar framework as it allows connections to be trialled and reformed at will according to the specific artistic goal in mind.
As previously noted, the creation of audiovisual relationships is highly subjective and, to a large degree, is influenced by each individual artist’s interpretation of the two media. This is in direct contrast to the ideas of the early Greek philosophers who sought to find an overarching equation that would describe a single fundamental relationship between sound and image. Although these early attempts appear to have been in vain, the subjective nature creates a near-infinite landscape of potential for artists to express their own unique voices based on their subjective interpretations of audio and visual characteristics. The digitisation of all media that occurred with modern computation has in the past few decades provided audiovisual artists with an extremely powerful tool to probe and explore the creation of new relationships. As new technology emerges and more of the world becomes digitised, the field of audiovisual art will continue to evolve and offer up new suggestions for linking and combining sound and image.