Within the practice of audiovisual art, forming interesting collections of audiovisual relationships is fundamental, yet I believe the most important creative process exists in how those relationships behave and evolve over time. One analogy is to see the creation of audiovisual relationships like a composer selecting instrumentation for an orchestral work. The interplay between the instrumentation
emerges from collections of overlapping and intersecting musical patterns which the composer uses to create the composition. Likewise, an audiovisual artist can utilise pattern-generating modulation sources to animate the behaviour of the relationships. In doing so, evolving phrases are generated which the composer can use to transfer meaning to an audience.
The grain reordering patterns described in section 4.3.1 and the various control strategies described in section 6.3 were developed to facilitate the above approach. Of course, while a single parameter that manipulates the audiovisual construct can be directly controlled, it becomes unfeasible for a single performer to manipulate more than two parameters simultaneously in a live performance. Powerful control strategies are thus necessary to help the performer control huge collections of variables simultaneously.
A great example is the amount of control the grain reordering patterns enable to manipulate not only AVGS but the FM synthesis engine and the visual shader synths. Through manipulation of the grain reordering patterns, the user is defining the reordering of audiovisual grains throughout time, and the value of FM synthesis parameters that are in turn directly driving the parameters of the visual shader synths. Consequently, the performer can focus on improvising with various rhythmical patterns that affect the global audiovisual state of the instrument.
Finally, creating banks of presets in the studio that can be later recalled during a performance captures the best aesthetic qualities of Kortex and creates confidence when performing. Playful experimentation in the studio allows the user to trial and tweak combinations of variable states until a desired aesthetic state emerges. Creating a preset then captures the global state of the instrument which the performer can then instantly recall and present to an audience. In this way, the user captures the best of the aesthetic states that emerge during the studio exploration process.
I believe it is important for this process to happen in the studio instead of in front of an audience, for two reasons. Firstly, it can take a long time to reach the ideal combination of variable states. A slight change in the value of a single parameter can dramatically change the resulting aesthetic. Secondly, an audience can be exposed to a larger number of aesthetic states that the performer has deemed interesting through recalling presets. Although the performer can explore variable combinations in front of an audience, they can at any time recall a preset to instantly provide contrast at pivotal moments in the set. The ability to slide between these states over time using tweening functions provides the performer with an extremely powerful and versatile strategy when performing live audiovisual art.
7.6 Technology
7.6.1 Hardware Performance Boost
Kortex was developed with, and performed on, an early 2011 MacBook Pro laptop for the first three years of my candidature. As the prototypes’ complexity increased I decided to change the internal hard drive to a 480GB SSD and to increase the amount of available RAM from 4GB to 16GB. The change from a traditional hard drive to an SSD contributed to a 20 fps increase in speed, while the extra 12GB of RAM meant I was able to store up to one minute of video frames in memory when using visual granular synthesis. The upgrade in speed and memory enabled me to add functionality into the prototype that would not have been possible otherwise. However, even with the upgraded performance, the MacBook Pro could only run the prototype at 60 fps with a resolution of 512x512 pixels on a single screen.
To enable Kortex to run at 60 fps using a triple-screen setup I built myself a custom hackintosh mini- ITX desktop computer (see section 3.6.2). With the powerful CPU and GTX Titan graphics card, I was able to develop Kortex while outputting to three monitors with a resolution of 1024x1024 at 60 fps. Furthermore, the size of the mini-ITX case allowed me to transport the computer as carry-on luggage when performing interstate and internationally. Considering the ease with which I am able to upgrade the machine with faster technology in the future, in comparison to a laptop, while retaining the convenience of portability, has led me to conclude that custom mini-ITX computers provide the best overall value for touring audiovisual artists.
7.6.2 Audiovisual Ring Buffer
The decision to utilise a circular ring buffer to store, read and overwrite audio and visual media into their respective buffers led to a several positive AVGS enhancements. Without a ring buffer, new content had to be uploaded into a temporary memory buffer frame by frame. Once all content was uploaded, the current audiovisual buffer was swapped with the temporary buffer and new content was then able to be manipulated. The process of uploading new content into the temporary buffer while the current buffer was being used required a considerable amount of CPU cycles, halving the frame rate of the playback of the current buffer. Furthermore, uploading new content with a length of 30 seconds would take on average 15 seconds to complete. This meant I suffered an unacceptable performance cost, and had to plan well in advance to evolve the audiovisual material. In contrast, the adoption of a ring buffer means only one buffer is required for reading and writing audiovisual content. This alleviates any performance costs when writing into memory and enables new content to be immediately accessible for AVGS processing.
Another advantage to this technique is the ability to stream live input from multiple sources in real- time. For example, capturing microphone and camera input, live television or streaming from YouTube expands the creative potential use of AVGS as a real-time processing technique. Finally, the ability to immediately select and process audiovisual content enabled a similar creative approach to real-time DJ and VJ performance software.
7.7 Summary
In this chapter I discussed the results and findings relating to key aspects of my research that emerged through using Kortex as a live audiovisual instrument. I showed that manipulating the size of the grain envelope produces different perceptual effects, specifically how the visual short-term memory phenomena produces illusionary overlaps when grain size is defined as a single frame. This was followed by examining the practical benefits of using both MIDI controllers and touch screens as performance interfaces to control live audiovisuals. I concluded that MIDI controllers excel at fine parameter control because of their tactical feedback, while touch screens excel at controlling functionality that requires visual feedback. Next, the creation of audiovisual relationships was shown to be mathematical, metaphorical/intuitive or intrinsic. I presented my personal observations about the subjective nature of creating relationships between image and sound and suggested that audiovisual artists use a modular assignment matrix to easily trial and form connections. Once an audiovisual relationship has been created, powerful pattern-generating control strategies are vital to define the relationship’s behaviour over time and to assist the performer in manipulating large collections of audiovisual parameters in real-time. I then discussed the real-time performance boosts gained from building a custom hackintosh mini-ITX computer. Finally, utilising a live audiovisual ring buffer technique was shown to expand the potential application of AVGS to include live streaming sources and strengthened the capabilities of AVGS as a real-time performance technique.
8 Conclusion
8.1 Overview
This chapter outlines how my research project has contributed to knowledge in my field of study and summarises my research outcomes in relation to the research objectives. My objectives included: (i) to establish a visual granular synthesis processing technique; (ii) to create a synchronised audiovisual software environment; (iii) to realise the creation and performance of audiovisual art in real-time. The significance of the research findings are presented, as well as their limitations. I give recommendations for future research and potential applications of my project instrument that emerged as a consequence of my research.