QEXIVMEPIW VIPEGMSREHSW '-*8%

This section discusses three systems that make use of gestures as a mechanism for interaction.

3.2.1 Gscore

The Gscore editor (Rubine, 1992), developed at Carnegie Mellon University, enables users to use gestures to place and manipulate notes on a musical score. It is an example of a system that illustrates the use o f a mouse to perform 2-dimensional (2D) gestures and two-phase interaction (an interaction that takes place in two consecutive phases). In this two-phased interaction, the first phase is the recognition of the intention to interact, while the second

phase is the actual manipulation. In the recognition phase a gesture, such as that made when a user stops moving the mouse while still pressing the button, is recognized. In the second phase the user can manipulate parameters interactively (for example, manipulating the upward and downward note stems). An example given by Rubine (1992) is that after the time-signature gesture is recognised, the x and y coordinates of the mouse interactively control the numerator and denominator o f the time signature.

Gscore supports the use of a variety o f gestures during editing, for example gestures exist for whole, half, quarter, eighth and sixteenth notes. Although it would have been possible to have a single gesture to represent all notes and interactively control the duration and stem direction of the note, the separate gesture option was chosen to provide faster interactions. However, it is possible to edit the set o f gestures and their meanings at runtime to try out various interfaces.

The creators o f Gscore also developed a two-phase multi-finger interactive environment at the same time. This is a drawing program where the interaction takes place in a Sensor frame (a frame with infrared sensors to capture finger movements in space). As this system was originally developed for interaction via a mouse, the mouse operations have been mapped onto single finger gestures. Once a gesture is recognised, the other fingers are used to control additional parameters. So, for example, after a line is created, the first finger rubberbands one endpoint o f the new line, and additional fingers control the line’s colour and thickness.

Gscore is an example of a 2D gesture-based system that can be used to create music. It achieves this by allowing gestures to be created using an existing HCI mechanism, i.e. the mouse. As discussed in chapter 2, though this is an interaction mechanism that is possibly familiar to the users, it is still artificial and is not an everyday human communication method. The second application might allow users to perform gestures with their fingers, but the gestures produced are merely mappings o f those usually performed with the mouse. For a more natural interaction approach, gestures should be performed with the hands.

Despite its shortcomings, Gscore does illustrate how gestures can be successfully incorporated into existing computing domains, such as music creation and drawing. This system is also a good example of two-phased interaction, which is the basis o f an interaction dialogue.

3.2.2 Digital baton

The Digital Baton is an interface for real-time gestural control. It was originally designed by Marrin (1997) at the MIT Media Lab, as an instrument on which to perform computer music. It achieved this by replicating as closely as possible the feel of a traditional conducting baton

and at th e sa m e tim e g iv in g th e u se r a c c e ss to a large n u m b e r o f in tu itiv e c o n tro l p a ra m e te rs th ro u g h a se rie s o f se n sin g sy stem s. M any o f th e c h a ra c te ristic s o f h an d m o tio n a re c a p tu re d and used fo r input an d c o n tro l o f d isc re te an d c o n tin u o u s fu n c tio n a litie s . F o r e x a m p le , th e d ig ital b ato n can e x e c u te e x a c tly -tim e d a c tio n s o f in d iv id u al n o te s and high level fu n c tio n s such as sh a p in g v o lu m e s and c o o rd in a tin g se p a ra te e v e n ts in tim e (M a rrin , 1997).

T h e se n sin g sy stem s used are an in fra red L E D tra c k in g sy stem , a c c e le ro m e te rs, an d p re ssu re se n so rs. T h e se se n so rs send c o n tin u o u s v a lu e s fo r th e b a to n 's p o sitio n , o rie n ta tio n , a c c e le ra tio n , and su rfa c e p re ssu re to a c o m p u te r via an e x te rn a l tra c k in g un it. M u sical c o n d u c tin g , b e in g a g e stu ra l la n g u ag e fo r m u sic, p ro v id e s a go o d initial fra m e w o rk fo r th e D igital B ato n , as it is a sy stem o f m a p p in g s b etw e en sp e c ific g estu ra l c u e s an d th e ir in te n d ed m u sica l resu lts. H o w e v e r th e se n sin g te c h n o lo g ie s im p lem e n te d in th e D igital B aton a llo w it to be also m a p p ed o n to o th e r in te ra c tio n m e c h a n ism s and th e sy stem a lso fe a tu re s a 3D m o u se , inertial g u id a n c e sy stem and m in i-k e y b o ard . F ig u re 3.1 illu stra te s th e D igital B aton in use.

Teresa M arrin

Figure 3. 1 - T h e D igital B aton (M a rrin , 1997)

T h e d ig ital bato n is fu rth e r e x a m p le o f a g e s tu re based sy stem and illu stra te s h o w a lte rn a tiv e te c h n iq u e s can be used to c a p tu re g e stu re s. In th is c a se , g e s tu re s a re p e rfo rm e d u sin g a sp e c ia lly d e s ig n e d c o n d u c tin g b ato n . It also d e m o n stra te s th e m a p p in g o f g e s tu re s to in te n d ed m u sica l resu lts. O f p a rtic u la r re le v a n c e is th a t it is a sy stem w h e re th e u se r d ire c ts sy stem b e h a v io u r th ro u g h g e s tu re s (m u sic in th is ca se ). In p rin c ip le , th is is sim ila r to th e ap p ro a c h p re se n te d in th is th e sis. H o w ev er, as w ith th e G sc o re sy stem th e g e s tu re s used w ith in D igital B aton are u n n atu ra l an d a re m a in ly fo cu se d on c o n d u c tin g m u sic and te c h n ic a lly rely on the use o f th e b ato n . W hat is in te re stin g w ith th e D igital B aton is th a t alth o u g h it w as in itially d e sig n e d as a c o n d u c tin g d e v ic e it has e v o lv ed into a d e v ic e th a t can be used as a 3D m o u se.

3.2.3 WorldBeat

T h e W o rld B e a t system is a tw o -b a to n g e stu re b ased sy stem , w h ich w a s d e s ig n e d for p e rm a n e n t d isp la y in th e A rs E le c tro n ic a C e n te r (A E C ) in L inz, A u stria (B o rc h e rs 1997). T h e A E C is a te c h n o lo g y “ m u seu m o f th e fu tu re ” . T h e p u rp o se o f W o rld B e a t is to d e m o n stra te to th e v isito rs th e p ro sp e c ts o f th e use o f c o m p u te rs in m u sica l c re a tiv ity an d e d u c a tio n , re g a rd le ss o f th e v is ito r’s p rio r c o m p u te r o r m u sica l k n o w led g e . T h is w a s a c h ie v e d th ro u g h a set o f m o d u le s, ea ch d e m o n stra tin g a d iffe re n t a s p e c t o f c o m p u te r use in m u sic. V ery b riefly , th e se are:

T h e Joy-Sticks m o d u le w h ich a llo w s th e u se r to p la y d iffe re n t “ V irtu al In stru m e n ts” .

T h e Virtual Baton m o d u le th ro u g h w h ich th e u se r m ay c o n d u c t a p ie c e o f m usic.

T h e Musical Memory m o d u le w h ich is a g am e w h ere th e u se rs h av e to re c o g n iz e th e in stru m e n t from its sound.

T h e NetMusic m o d u le w h ere th e u se rs can c o o p e ra tiv e ly play m usic.

T h e Musical Design Patterns m o d u le w h e re u se rs can c h a n g e th e b asic p a ra m e te rs o f p ie c e s o f m usic.

T h e id ea b eh in d W o rld B e a t is to e n a b le th e v isito rs to c o n tro l th e c o m p le te e x h ib it u sin g tw o in fra red b ato n s (see F ig u re 3.2). A ll m a jo r ta sk s c a rrie d o u t d u rin g th e in te ra c tio n w ith th e e x h ib it are in te g rate d into a sin g le in te rfa ce .

T h e v isito rs can use th e tw o b ato n s to in te ract w ith a g rap h ic a l u se r in te rfa c e , fo r e x a m p le th e v irtu a l b ato n c o n d u c tin g m o d u le , an d th e n use th e b a to n s to c o n d u c t th e p ie ce . T h is is th e m ain d iffe re n c e b etw e en W o rld B ea t and th e D igital B aton p re v io u sly d isc u sse d , as th e d ig ita l b ato n is used o n ly fo r c o n d u c tin g p u rp o se s and not n av ig atio n .

In order to operate the system the user stands in front of the exhibit and, watching the computer monitor, gestures with a baton in each hand (see Figure 3.2). As each baton contains infrared light emitting diodes, infrared signals are constantly emitted in all directions. Spatial signals giving the position of the baton in the form o f x y b (x for horizontal position, y for vertical position and b for button press) coordinates are sent when the action button is pressed or released, while the infrared tracker, which is mounted directly below the monitor, measures the angles at which it receives the signals from the two batons. The data stream of the coordinates of the batons is sent to the base unit, where it is converted into MIDI messages. Basic gesture recognition is also incorporated in the system, which recognizes “beat” gestures as notes. This feature is used in the modules where the user plays virtual instruments using the batons as drumsticks. All events that describe MIDI playing messages are finally sent to the base unit where the requested audio signals are created and then are sent to the amplifiers, speakers, tape desk or headphones.

The right baton also functions as a pointing device, essentially acting like a 3D mouse for all navigational requirements o f the system. So the user may navigate through the application by pointing at the yellow on-screen spots with the right baton and pressing the action button. However, apart from deictic gestures, other types of gesture are also supported by this 2-baton system. In the Joy-Sticks modules the batons are used as mallets to perform gestures commonly used to play instruments such as the drums and the xylophone. In such cases a natural mapping of the downward “beat” gestures are used to play the instruments in a velocity-sensitive way. All other interactions are again cases o f “beat” gestures.

WorldBeat is a further gesture-based system that features batons. It can be used as a more general use interface for creative activities, as shown in the Joy-Sticks module. This system requires no prior knowledge o f music or computing and has a very small learning overhead. This makes it attractive to the users, who are mainly children, and its usability helps attain the goal o f bringing museum visitors closer to music. This system is restricted to music applications, mainly because o f the use of conducting batons. The main drawback o f this system is that while using a baton in the context o f music might seem a natural way of interacting, it is still restrictive in that it is less suitable for use within other domains, such as History or Science, where such a mechanism is not so obviously meaningful.