Capítulo I Reflexiones Sobre la Historia del Violín.

The sketches presented in this section explore the integration of elements of physical interaction in the design of affordances for direct touch interfaces on interactive surfaces.

Chapter 3 reviewed a number of research projects addressing the manipu- lation of digital information on interactive surfaces. Several of these examples adopt a stylus or an interactive object as transducers. These enable the de- sign of manipulation vocabularies that are based on the gestures afforded by the transducers themselves (e.g., crossing-based and marking gestures for pen input (Accot and Zhai, 2002); tilting and rotating gestures for input with tangible tokens (Rekimoto and Sciammarella, 2000)). Other work exploits the richness of hands and multi-finger manipulation in the physical world to design a set of gestures for interacting with digital information (e.g., (Wu

4.2. Reflecting on 2D Surfaces

et al., 2006)). Such a vocabulary of gestures mostly needs to be learned and remembered by the user in order to interact with the system.

In this context, emerges the question: How can we display digital infor- mation on a surface in such a way that it suggests users what gestures to perform, thus minimizing their learning effort? An hypothesis is that if we represent digital, abstract information in such a way that it metaphorically represents a physical artifact, we can design cognitive affordances for users’ gestures by relying on the physical affordances and manipulation vocabulary of the metaphorical source (cf. Chapter 2, Section 2.7, for a distinction of physical vs. cognitive affordances).

While designing cognitive, visual affordances for digital information in 2D, two main design aspects need to be addressed. On the one hand, visual cues of the displayed information can suggest its pliancy, i.e., its characteristic to be interactive (Cooper and Reimann, 2003). For example, frames can define semantic areas of interaction on the screen; 3D effects, colors, and shadings can bring the information in fore- or background and suggest information status (e.g., active/inactive). These visual cues provide what Pirolli and Card (1999) name information scent, which can enhance perceptual as well as cognitive viewers’ processing of information: I.e., they can accelerate users’ visual “scanning” (Nielsen, 1999) of the information display.

On the other hand, the metaphoric link to real world objects and to their affordances in the physical world can provide rich material for the design of affordances for the manipulation of digital information. The virtual repre- sentation of a lever doorhandle, for instance, can be mapped to the natural gesture that we make when we push down a real physical lever. The rep- resentation of a steering wheel can be mapped to the turning gesture we perform while driving. In these cases, the mapping relies on the analogy between the elements representing digital information and the affordances of their referents in the physical space. In this respect, as designers, we need to think thoroughly about a manipulation vocabulary (and the representa- tion thereof) that is suitable for the type of transducer used for input: e.g., multi-finger or single input point, such as in the case of pen input.

To demonstrate this concept, and building on the considerations above, these ideas were explored through the design (or sketch, rather) of a metaphor- ical interface for multi-finger hand gestures on a wall display. The next para- graph illustrates the design of such an interface.

4.2.1

The Mug Metaphor Interface

The underlying idea of this sketch of interaction was to explore how one can map the affordances of real world objects to gestures, relying on the

4 Exploring Affordances for 2D and 3D Manipulation through Sketches

Figure 4.1: The figure illustrates the idea of metaphorically mapping the physi- cal affordances of real world objects to cognitive affordances for gesture-based manipulation of digital information on interactive surfaces.

conceptual model in which such real objects are embedded. Referring to the qualities of physical interaction identified in Chapter 1 and represented in Fig. 1.3, the Mug Metaphor Interface explores the integration of the following aspects:

• Metaphorical representation;

• Directness;

• Continuity of action;

• Two-handed cooperative work.

The Mug Metaphor Interface relies on the physical affordances provided by a real mug, and metaphorically represents it as a container of information. When manipulating a real mug we know we can move it around by holding its handle and incline it to pour its content (see Fig. 4.2, b and c). Empty mugs are expected to be lighter than full ones (e.g., contain less data); steaming mugs are expected to be hot (e.g., contain recent data). Additionally, a mug is a mundane object which we use in different environments, e.g., in the office, in a living room, in a kitchen: Thus, the metaphor is not strictly related to the office domain, and can be valid in others as well.

In such a concept, mugs and units of information - the latter represented as kind of drops - can be manipulated across the display. Pie menus appear in

4.2. Reflecting on 2D Surfaces

correspondence of the hands, thus “following” the user while moving across the display, rather than being operable just in a fixed location on the screen. This responds to the need of freedom of movement of the user, and to enable two-hands cooperative interaction. The dominant hand, e.g. the right one, is devoted to the manipulation and navigation of information. A pie menu displaying containers of information is displayed in correspondence of the right hand (see Fig. 4.2, a). The non-dominant hand (e.g., the left one) works as command invocation, managing a menu of resources (e.g., drain, displays, and printers). Such a menu appears when the non-dominant hand double-taps the interactive surface. The circular menu can be scrolled with a movement of the finger on a holed gear, which makes the circle segments rotate (see Fig. 4.2, d). The dominant hand moves units of information to the preferred resource (e.g., to cancel an information unit as in Fig. 4.2, e).

Figure 4.2: The Mug Metaphor Interface: The figure illustrates different stages of the mock-up realized in Flash.

4.2.2

Observations and Implications

The mock-up of the Mug Metaphor Interface was realized in Flash according to a storyboard of possible manipulation tasks, such as for example “open”,

4 Exploring Affordances for 2D and 3D Manipulation through Sketches

“close”, “move”, and “delete”. Such an interface was informally evaluated in our lab, both with colleagues and with people who are not familiar with this research and with the technology it adopts. We first demonstrated the set of tasks we had implemented in the Flash interface on a SmartBoard and mimicked the gestures we wanted to suggest (e.g., rotating the mug for opening; dragging the mug handle for moving; tapping the screen surface with a hand in a certain place and time sequence for invoking the pie menu) so as to show that different gestures were possible for different tasks. We then asked individual users to perform a certain set of tasks such as “explore the content of the mug”, “move the mug”, “cancel the information unit”, “scroll the menu”, and so forth, in such a sequence that could be supported by the mocked-up storyboard.

The participants easily interacted with the interface and could accomplish the tasks. It was curious to observe, though, that despite the mimicking demo in the introduction, they seldom used multiple fingers for interacting with the surface. In order to open or close the content of the cup, for example, some users were just quickly tapping on the body of the cup itself with their index and middle fingers (i.e., similarly to a mouse click gesture) instead of continuously rotating it with the whole hand.

In a Flash mock-up it is not possible to implement, obviously, the use of multiple fingers as a constraint: Thus, the graphical output behaves accord- ing to a single input point, such as tapping. The implementation of other constraints in the software could have possibly influenced users’ interaction behaviors in a different way. This simple mock-up, though, was useful to explore potentials and limitations of the design ideas, and suggested that some issues about users’ expectation of digital behaviors in hybrid interac- tions should be investigated further and more systematically (cf. Chapter 7). Additionally, it inspired the idea that if one was able to implement or simulate haptic feedback to the action afforded by a visual cue in the dis- play, this fact could possibly affect users’ behaviors and learning process: E.g., the force to be applied on a cup in order to incline it could be coupled by a haptic resistance; full cups could be harder to move than empty ones. Such a conceptual model could be reinforced by audio cues (e.g., the sound of splashing water when the content of the cup is poured) so as to provide sensorial affordances (cf. Chapter 2, Section 2.7).

In this sense, the multimodality of interaction with digital information can potentially provide more analogue interfaces. Muscular memory (rather than purely cognitive one) could then be exploited by users to structure frequent, patterned tasks into stylized or abbreviated gestures (such as is the tenet of crossing-based gestures, cf. Chapter 3, Section 3.1.1), thus providing us, as designers, with the possibility to build a gesture vocabulary that exploits