Equipamiento

KSFT system 4.3.2.1 Introduction In Section 1.2.3, we discussed the

electromechanical playback of stored sound fragments using gramophone record players, or turntables. We also discussed how electronic playback followed, using digital samplers. We then discussed sampling independently of specific hardware, in Section 1.4.2.3, as a form of computed sound (see Figure 1.10).

The human control actions that came into use with turntables included scratching. Here, the fingerpad is pressed down on a flat contact surface, after which surface- parallel fingertip movements are performed. During this, fingerpad-parallel forces are applied via the contact surface (see Section 2.3.1). The causal relationship enabling instrumental control of musical sound, here, is that for the given stored sound fragment, spatial fingertip position is made to correspond to temporal playback position.

Like the examples in Section 4.2.2.1, this setup enables display, via touch, of the state of the sound-generating process. But again, we may envision touch display that might better inform control actions, by providing higher levels of detail: As the signal variations in the record groove do not result in felt forces, touch display during scratching is not specific to the actual stored sound fragment being played back.

Therefore, below, we will explore display that is more specific to the stored sound

fragment. We will use the KSFT system to implement friction output during active touch that reflects ongoing stored sound fragment playback with millisecond resolution [De Jong 2010b].

4.3.2.2 The sound-generating process As before, we use a time t∈ , based on aℕ

constantc_{sampling rate}(Hz). The sound fragment, then, was represented by a series of

amplitude values within a [-1, 1] range, stored as a one-dimensional array in a memory

buffer⃗b_{stored sound fragment}. Playback was characterized by:

o_audio[t] = f_{index and interpolate}(s_{sample index}[t],⃗_b

stored sound fragment)

Here, f_{index and interpolate}calculates the amplitude value for the specified index, which

may be a non-integer number: cubic interpolation is performed to enable intermediate values. A delay of 1.0 ms was applied to the resulting signal, to optimally synchronize audio output with related friction output (see Sections 3.3.1.3 and 3.3.1.5).

4.3.2.3 Computed touch Friction output displayed stored sound fragment playback:

sμ[t] = 1 n

∑

i=n−1

∣

∣

This withn = c_{sampling rate}×0.001 and fromt ≥ n−1, so that sμ[t]tracked the

average absolute amplitude over the most recent millisecond of audio output. Then, to better match the perceived loudness of audio output, friction output was not computed directly from the average absolute amplitude, but via:

o_friction[t] = f_{dB to N mapping}(20×log10(

s_μ[t]

c_ref ))

This was done with reference amplitudec_ref = 1, so that the maximum absolute

amplitude corresponded to 0 dB input to the f_{dB to N mapping}function, which clipped and

4.3.2.4 Instrumental control of musical sound The causal relationship enabling instrumental control was:

s_{sample index}[t] = i_{x displacement}[t] × c_{samples per mm}

Here,i_{x displacement}[t](mm) is the displacement input for the x planar direction of the

KSFT system (see Section 3.3.1.1). The constant c_{samples per mm}was varied between

algorithm runs.

4.3.2.5 Evaluation When using appropriate values forc_{samples per mm}, the algorithm

described in Sections 4.3.2.2 to 4.3.2.4 enabled instrumental control similar to turntable scratching, with sideways fingertip movements controlling variable-pitch playback of stored sound fragments. The algorithm was tested in a pilot experiment with 5 volunteer test subjects [De Jong 2010b]. The subjects were presented with a stored sound fragment consisting of silence, followed by 240 cycles of a sinusoid, followed by more silence. Sine cycles were identical, of maximum amplitude, and had a length of 100 samples each. The test subjects were asked to explore the stored fragment, by using sideways fingertip movement at different speeds. This was done repeatedly, for the following parameter configurations:

•c_{samples per mm}= 4000, 500, or 8 samples/mm.

During exploration, the abovec_{samples per mm}options resulted in frequency ranges for

audio output of [104, 13360] Hz, [13, 1670] Hz, or [0.2, 26.7] Hz, respectively. Each of the three options was presented twice: first without, then with friction output enabled in software.

Forc_{samples per mm}= 4000 samples/mm, and forc_{samples per mm}= 500 samples/mm during fast movements, audio playback of the sine wave fragment coincided with a heightened friction level, computed from the averaged signal intensity (see Section 4.3.2.3). Here, test subjects clearly felt that sound and touch were related, both resulting from movement in or over the same specific surface area.

The algorithm was publicly demonstrated at the 2010 international workshop on Interactive Sonification in Stockholm, Sweden. There, participants also could record the stored sound fragment live via a microphone, to then explore the result as in the experiment (see also Figure 4.2).

The results of the pilot experiment indicate that computed friction output can be used to add touch display during scratching that is more specific to the stored sound fragment being played back. This can happen “isospatially”: with temporally coinciding features in sound and touch being perceived as belonging to the same spatial location. This suggests a potential use of computed friction as an aid to spatial orientation before or during fingertip control actions.

Figure 4.2 Movement speed input, audio output, and friction output during 5 seconds of instrumental control of a stored sound fragment. In the left section of the figure, a signal feature is first played back in one direction; then, in the middle section, this is done in the opposite direction. Finally, with fingertip movement having returned to somewhat before the initial position, in the rightmost section part of the feature is again played back, but now using a slower movement.