Las Tácticas en la Práctica

In the previous sections we gave a detailed overview of available hardware approaches – both from the literature (Section6.1) and our own approach (Section6.2). This detailed discussion of hardware solutions might seem excessive for a thesis with the goal of proposing a new model for

6.3 Chapter Summary 93

tabletop interaction. However, the author is convinced that tabletop computing is a sub-field of human computer interaction where an in-depth understanding of available hardware choices and understanding of new developments is crucial. Never before (maybe except recent developments in mobile phone user interfaces) were hardware and software developments so interwoven as they are in interactive surface computing. For example, consider the recent transgression from single- to multi-core CPUs. While this new hardware paradigm has significant impact on the performance and also on programming paradigms – the direct impact on how users interact with a multi-core system versus a single-core equipped system has not changed (at least not as a direct consequence). In contrast, tabletop hardware advancements (or limitations) influence the way humans can interact with the digital directly and inevitably.

This begins with the size of the table itself as Ryall et al. [RFSM04] observed in a study comparing group performance in identical tasks performed on differently sized versions of the DiamondTouch table. But it also matters how an image is produced; while front projected systems allow for table designs that have enough space underneath the projection screen for multiple users to sit around the table and rest their legs underneath the table, they suffer from occlusion problems due to objects in the optical path (e.g., hands, heads). Back projected systems mitigate the occlusion problem nicely but require more space under the projection screen which makes them more suitable for settings in which people stand around the table. This again has an impact on how people interact with a system. One can imagine that standing around a table would encourage more lightweight interactions while seated arrangements could allow for longer more in-depth work such as reading or drawing. Finally the sensing capabilities of the chosen approach may be most important aspect. The interaction possible on a system that only recognizes one or two touch points or pens does not lend itself toward the same interaction techniques as a system that is capable of sensing multiple fingers, whole hands and other physical objects simultaneously. Equally important might be whether a system can recognize multiple contacts, like many vision- based systems, or multiple users such as theDiamondTouchtable.

The influence is not unidirectional; while new developments appear in the hardware arena at a fast pace we as a community also continuously refine what interacting with interactive surfaces means. Therefore, we constantly refine the requirements for the design of tabletop hardware and impose new challenges for the designers of such hardware. For example, only a few years ago many doubted that being capable of sensing multiple points of contact was indeed necessary or beneficial (most techniques to do so are known for decades). The onset of products (e.g., Microsoft Surface, iPhone,DiamondTouch) that integrate multi-touch capabilities and graphical interfaces specifically designed for such hardware helped to create a new perspective on this issue.

94 6. Technical Foundation

The following table summarizes some of the sensing aspects to consider when choosing a hardware category or platform:

MT User ID Shape Information Proximity Information Object Presence Object ID E m b e d d e d C a p a c it iv e Diamond

Touch ü ü Bounding Box û û û

SmartSkin ü û ü ü ü2 _û O p ti c a l ThinSight ü û ü ü ü ü V is io n -B a s e d D ir e c t Visual Touchpad ü û û ü û û Play Anywhere one !inger per hand û Motion ü ü ü In d ir e c t FTIR ü û ü û û û Microsoft Surface ü û ü1 ü1 ü ü DViT 2 max û û û ü û

1: requires special software. 2: requires special tags.

Figure 6.17: Overview over most important sensing aspects of systems discussed in previous section.

Table6.17gives an overview over the systems discussed in the previous sections and how they perform in respect to the factors listed above. Equally important for the resulting user experience are the output capabilities of tabletop systems:

Occlusions: the display may be occluded or have shadows cast upon (e.g., front or top projected systems suffer from this problem)

Lighting: is the display still legible under various lighting conditions (projected and vision- based systems will usually work better under controlled lighting conditions)

Viewpoint independance: does the display support a large field of view and is it orientation independent (viewable from all sides)?

Resolution: is the display capable of displaying information at a reasonable resolution (e.g., impacting text legibility) - LCD based solutions will usually fare better.

6.3 Chapter Summary 95

Clearly these novel hardware developments enable new interaction styles that go beyond the cursor based interaction we are used to from regular desktops. Furthermore, our own explorations of digital tabletop interaction were severely constraint by the limitations of the input hardware, especially when considering our criteria offlexibilityandphysicality. After discussing the sensing side of the various approaches presented in this chapter, enabling the sensing fidelity we deem necessary, in the following chapter we introduce our new model for tabletop interaction that makes use of the rich kind of input data provided by vision-based interactive surfaces.

Chapter 7

Bringing Physics to the Surface

Up to this point we have discussed existing approaches to tabletop interaction. From the literature we have distilled two particularly popular interaction styles; gesture-based and tangible interac- tion. Studying our own prototypes based on these paradigms in Chapters 3, 4 revealed several issues and short-comings of these approaches. Foremost, the lack of flexibility which sometimes frustrated users. We have also observed that limitations inflicted by the hardware was a limit- ing factor in many situations. We could clearly observe that, especially novice, users wanted to interact with the system in richer ways than what could be sensed.

In Section 6.2 we proposed our own tabletop prototype capable of sensing rich user input. Furthermore, both the research community [MR97,Rek02, Wil05, Wil04, Han05] and industry [Mic08,DL01,SMA09] have demonstrated and sometimes productized setups capable of similar sensing fidelity. In this Chapter we want to explore how this rich sensing data can be applied to novel interaction techniques and ultimately propose a new model for tabletop interaction.

In our earlier explorations (cf. 3.5) we could observe how the more physical inspired gestures such as flicking of post-its were received more favorable (and memorized better) than the iconic, abstract gestures. We have also experienced how the scripted nature of these “pseudo-physical” interactions can be problematic. Users that were under the assumption that virtual objects behave according to the laws of physics or at least similar to real world objects often tried to perform other physical manipulations – these had not been anticipated by the developer in advance and as result were not possible. Therefore the main goal of this chapter is to develop an interac- tion model that allows for interactions with the virtual realm through manipulations similar to those performed on real world objects without the limitations imposed by scripted or pre-defined behavior of on screen objects.

Recently, sophisticated physics simulation packages have become accessible and found widespread use in many 3D computer games. These physics simulations are capable of mod- eling complex mechanical structures and model their behavior realistically in 3D graphical ap- plications. In our interaction model we make use of the capabilities of these physics simulations and explore the intersection of rich surface input data and virtual worlds augmented by realistic, open-ended and non-scripted behavior of virtual objects.

98 7. Bringing Physics to the Surface

Modeling rich 2D sensor data within a 3D physics simulation is non-trivial and we highlight and discuss several of the challenges we encountered when developing our interaction model. Based on our previous experiences we wanted our model to support the following aspects:

• _{Enable rich physical gestures through manipulations similar to the real world.}

• _{Using sophisticated physics simulations to add real world dynamics to virtual objects and}

enable users to interact through the exertion of forces such as friction and collisions.

• _{Support of multiple simultaneous contact points (not just fingertips) and and also real (tan-}

gible) objects.

• _{A technique that works within the bounds of the physics simulation and makes use of the}

sophisticated constraint solver build into many available software packages.

• _{A generic model that works with different virtual objects irrespective of their shape or}

material (e.g., boxes, spheres, cloth)

This enables a variety of fine-grained and casual interactions, supporting finger-based, whole- hand, and tangible input. We demonstrate how our technique can be used to add real-world dynamics to interactive surfaces such as a vision-based tabletop, creating a fluid and natural experience. Our approach hides from application developers many of the complexities inherent in using physics engines, allowing the creation of applications without preprogrammed interaction behavior or gesture recognition.

Contribution Statement:The evolution of the interaction model described in this chapter has been published as peer reviewed full paper [WIH+08]. Most of this work has been done at two Microsoft Research labs in Cambridge, UK and Red- mond, USA. Many of the discussed models have been designed and implemented while I was an research intern in the Cambridge Lab. Together with my man- ager Shahram Izadi I designed and implemented a version of the direct forces (cf. 7.2.1), joints (cf. 7.2.2) and discreet-proxy (cf. 7.3.1) model. The particle proxy

model was designed and implemented by Andy Wilson in Redmond (initially in- dependent). Armando Garcia-Mendoza re-implemented large parts of this work to produce a final version incorporating all variations of the technique. All co-authors contributed to the design, execution and evaluation of the study as well as to the submitted paper.