The context of the participatory workshops requires us developing a toolkit that could be used by participants to design their interactive sound prototypes based on
the recollection of sonic memories. The pilot study in Chapter 3 provided us with a technological component, based on the miniaturised accelerometer and a modular sound programming environment revealed to be functional and efficient for realising gestural-sound mappings. The anonymous and minimal aspects of the motion sensors did not show the same degree of physical and cultural affordances of control devices available in the market with built-in accelerometer, such as wiimote and smartphones. This shared consideration between the user study from 2012 and the pilot study conducted in Chapter 3, made the system an ideal candidate to be used in participatory design workshops as it could be easily attached to the body of participants. The specific idea for the workshops though relied on exploring sonic experience from participants, relying on their memory. Therefore, rather than us choosing the sound stimuli as in the pilot, we needed to allow participants to choose their own sounds. This would also apply to the choice of associated gestural mapping. The need of such modularity therefore developed in the concept of the development of a toolkit, which could help participants to choose their own sound and mapping.
Although the SID sketching tools reviewed in 3.3.5 proved to be useful for the workshops, some of their limitations – unease of use, lack of availability and being device-specific - were too inhibiting for adopting them in Form Follows Sound workshops. In our case, we needed a toolkit that could exploit participants’ embodied interaction and familiarity with the sounding and movement abilities of the body, that could be easy to use, non-device centric in terms of shape and manipulation and importantly available to a bigger community of non-specialised users.
The Gestural Sound Toolkit is our answer to this emergent need of the research. It is a software/hardware prototyping platform we designed to aid the development of interaction scenarios into working projects. The toolkit was developed collaboratively between Baptiste Caramiaux and myself. We designed the general idea and the graphical elements of the interface. On an individual level, I focused on the high level design of sound synthesis modules and the interfacing with the accelerometer-based devices we used in the workshops.19 Caramiaux developed the
movement analysis and machine learning modules and the low-level elements of the sound synthesis engine. The toolkit uses MuBu lib developed by the ISMM
team at IRCAM20 and CNMAT Max/MSP Externals library. 21 Both these libraries
are currently free to download and to use. Our toolkit can be downloaded for free.22
The function of the toolkit is to give participants a sketching tool to develop prototypes using interactive gestural-sound mappings. It integrates complex techniques for sound synthesis with machine learning of movement, making these techniques available to participants with no background in interactive systems, sound design, and more generally in programming and physical computing. It comprises modules for receiving movement data from sensors, analysing data through machine learning and gesture recognition, and mapping participants’ gestures to sound synthesis.
The toolkit consists of three main categories of modules. The first one is the Receiver module. It receives motion data from the Axivity wax3 wireless accelerometer device we will use during the workshops. It accepts any list of three values, such as in the case of Open Sound Control messages (OSC) that these devices – or any others compatible with this protocol – transmit.23 It also offers
possibilities for fine-tuning of the calibration system.
The second category of modules performs movement and gesture analysis (Analysis modules). These modules analyse accelerometer data such as reducing noise, extracting energy and impact, and performing gesture recognition. This is based on machine learning techniques, and it specifically uses the Gesture Variation Follower algorithm developed by Caramiaux (Caramiaux, Montecchio, et al. 2014). This is based on a learned, pre-recorded database of gestures and permits the early recognition of a live gesture as soon it starts. It also estimates variations of characteristics of speed, scale and orientation. This is particularly useful in the case of real-time interactions with sound in the prototyping phase as it facilitates procedures of gestural-sound mapping.
The Synthesis modules compose the third block of our toolkit. They enable participants to play with pre-recorded sounds and to manipulate them. In the Trigger module participants can play sound samples once, as if pressing a key on a
20 MuBu lib is available at http://forumnet.ircam.fr/product/mubu-en/
21 CNMAT Max/MSP externals are available at http://cnmat.berkeley.edu/downloads 22 Gestural Sound Toolkit page on Github. Fork of the author based on collaborative work with Dr Baptiste Caramiaux, who is currently maintaining it:
https://github.com/12deadpixels/Gestural-Sound-Toolkit
keyboard or hitting a drum snare. The Scrubbing module allows users to start the sound playback from a chosen playhead position and to change it in real time. The Manipulate module controls frequency aspects of the sound, the pitch, speed and filtering. The Scratch module works similarly to a vinyl player and the variation of speed changes the pitch of the sound sample. The Stepping module divides the sample in parts according to the detected greater variation of amplitude of the sound, such as for beat detection. These can then be played as single triggers, creating a sort of shaker effect.
The modules can be assembled and linked as the users prefer. They are individual and can be copied, duplicated and rearranged. Figure 4‑A shows an overview of the available modules, while Figure 4‑B shows an example scenario.
Figure 4-A: An overview of the modules available in the Gestural Sound Toolkit
In our workshops we introduce the toolkit with a preliminary tutorial, including a short presentation and ending with a short hands-on session where participants build simple interaction examples, such as triggering a sound with an impact gesture captured by the accelerometer, or shaking a sound with rapid iterative movement. The objective for the tutorial and the following short hands-on session
is to provide participants with enough knowledge about the toolkit to develop the final projects on their own. The effort we put into designing and programming the toolkit aimed to make it easy to use and effective for general users to quickly prototype gestural-sound mappings. The workshops offered us a possibility to test this toolkit in action with non-expert users.
Figure 4-B: Gestural Sound Toolkit: Articulating sound through an impact gesture. Accelerometer data enters (top), energy is calculated (middle), and a kick is generated (bottom) when energy
crosses a threshold, resulting in sound being triggered (right)