Balance de Capitulo

pixel intensity based depth estimation in our first setup. The second tabletop configuration shows how in-the-air interactions can be facilitated using a holographic projection screen and a per-pixel depth sensing camera.

In addition, various rendering techniques have been proposed to provide better feedback when interacting with 3D data on the surface from above the surface. We have also showcased some of the interactions enabled by our techniques. Clearly we have only begun to explore the possibili- ties unlocked by the gained interaction fidelity. While we have explored the high DOF interaction techniques in a literal context – that is 3D input and 3D data – we are convinced that our, and similar, techniques can be useful in a broad range of applications including 3D and (extended) 2D applications.

10.2

Conclusion

Going back to our problem statement presented in Section1.2 the main goal of this thesis was to define a new model for tabletop interaction. When launching into the research for this disser- tation many hardware and software solutions had been proposed and studied such as pen-based, multi-touch and tangible interaction. These interaction techniques, however have often been de- signed in an ad-hoc manner and studied in isolation. We wanted to gain a better more structured understanding of the design space. In order to achieve this goal we followed a threefold approach. First: a literature analysis with special focus on tabletop interaction styles. Second: explorations into the identified two main interaction styles (gesture-based and tangibles). Third: proposal, evaluation and refinement of our own model for tabletop interaction.

In order to structure and focus our investigation of tabletop interaction styles we aspired to provide answers to the following two questions:

What is the Interface? Real world interactions benefit from tactile and other rich sensory

feedback. As well as from our aptitude to manipulate objects in various ways. Thisphysicalityis only poorly represented by most direct-touch interfaces. Often it is assumed that digital tabletops are inherently flat and made of glass or acrylic. Also it is often assumed that multi-touch simply means having the equivalent of multiple cursors controlled by several fingertips simultaneously. In this thesis we have shown that interacting through direct-touch carries much richer meaning and interfaces for this emerging class of computing devices. It may be required to think of user interfaces for digital tabletops not as an evolution of the standard desktop interaction paradigm, but one that has a whole new set of unique parameters.

In PartIIwe have seen that both tangible and direct-touch based interaction have promising aspects and limitations as interaction styles. Both promise a natural and graspable interaction experience, both are accredited with a low learning threshold and both offer support for co- located collaborative work and social interaction. Our aim was to explore this rich design space as a whole in order to provide a better understanding how to best use them. We would argue that our criteria offlexibilityandphysicalityhas been helpful to generalize some of the findings

158 10. General Discussion and Conclusion

specific to our implementations to a broader understanding of the design space. Especially with regards to uncovering the limited flexibility afforded by these interaction styles.

In particular we have seen that physical aspects play an important role in interactive surface computing and can be supported through a variety of means. For example, those explored in this thesis such as real world metaphors in the UI, tangible objects as well as virtual objects that behave more realistically. Another way to provide users with physical affordances are means to create richer feedback at the interface level be it through passive haptic feedback – partially explored in our malleable interactive surface prototype (cf. Section 6.2 and in the literature [VMK+05, KVM+05, Sin97, SGHB07] – or through active feedback via actuated tangible ob- jects [PMAI02] or via dynamically changeable physical buttons on direct-touch enabled surfaces [HH09a].

We have also seen that is important to compliment this physicality with a great deal of flexi- bility in the interface. In our explorations it has been especially problematic combining pseudo- physical behavior and appearance with scripted and therefore constrained interaction techniques. Our new model for tabletop interaction has shown ways to create more open-ended and flexible interaction techniques that allow for re-purposing and appropriation of user interface elements through rich parameters such as friction, velocity and collisions known from the real-world. We have also seen that this model was well received by users in a lab-based study. Clearly we have only begun to explore this interaction model and many questions remain unanswered as of now. For example, we have yet to build applications based on this model that go beyond literal translations of object positions. We also have to consider how many aspects that go beyond the physically possible can be modeled in with our approach. For example, copying, scaling and zooming of documents is not very well represented in our model. In general it is worthwhile considering how this model could be crossbred with more abstract, traditional interaction models such as the WIMP paradigm.

Where is the Interface? Because tabletops have planar 2D displays it is often assumed that

applications and interaction should be 2D as well. Interaction with such surfaces is inherently constrained to the planar, 2D surface of the display. For many tabletop interactions this constraint may not appear to be a problem, particularly when direct manipulation with 2D content is de- sired. Recent research is however beginning to motivate the need for rendering 3D content on tabletops [HCC07,HC07]. Also during the evaluation of our own work on physics enabled table- top applications [WIH+08] it became apparent that because the interaction – the sensed input and the corresponding displayed output – is still bound to the 2D surface, there is a fundamental limitation in manipulating objects using the third dimension.

We have outlined in Chapter 8 that emerging display technologies can enable sensing that goes beyond these 2D limitations. In Chapter 9we have started to explore the space above the table as interaction space with equal value as the display surface. While we haven’t fully ex- plored this space we are convinced that tabletop setups that allow for continuous interaction on

and above the surface are a compelling area for research and promise to be useful in many ex- citing application domains such as architectural or medical imaging but also 3D gaming. In the light of our initial exploration we would argue that we have done an initial step toward blurring

10.2 Conclusion 159

the line between the (interactive) display and the environment it exists in. We think it is worth- while thinking of the entire space surrounding interactive devices as potential interface with the digital. This could involve physical (inherently 3D), possibly actuated, objects in combination with tabletops supporting advanced sensing and feedback mechanisms [IHT+08,KN08, Wil07] or via rich natural hand gestures sensed in-the-air as discussed in Chapter9.

160 10. General Discussion and Conclusion