Las tácticas de los maestros como elementos esperanzadores en la construcción de

Interacting with virtual objects rendered on the surface in this way opens up 4DOF interaction capabilities on and above the surface. However, there is also a key challenge when facilitating

Figure 9.4: Using the combined depth sensing and pinch to place an object into a virtual container. Height of virtual objects is directly controlled by distance of user’s hand to the screen.

136 9. Interactions in the Air - Adding More Depth to Interactive Tabletops

Figure 9.5: Depth Shadows are conceptually shadows cast from the user’s hand into the 3D scene to allow closer coupling between input and output spaces.

this type of interaction – the user’s hands and rendered content are only in direct visual contact when interacting directly on the surface. A key challenge arises in the loss ofdirectnesswhen a user moves from interacting on the surface to above it.

Returning to our earlier discussion around perceived display space, it becomes clear that we are conceptually creating a 3D volume inside the tabletop. Clearly the user’s hands sit outside of this virtual space, separated by the actual physical bounds of the display. Using raycasting and the virtual joint metaphor means that users have even a greater sense that their hands are decoupled from the 3D volume rendered on the tabletop. To try and compensate for this decoupling, we describe a shadow-based technique that helps connect the user’s hand in the real-world with the objects in the 3D scene. We do this by conceptually casting a shadow of the user’s hand into the 3D scene, fusing this with the shadows cast by the virtual objects in the scene. This provides a real-world metaphor to map between actions in the physical space and interactions inside the virtual 3D scene, as shown in Figure9.5. These shadows can also function as additional depth cues for the user when adjusting an object’s position along the Z-axis.

We use a GPU based shadow mapping approach to create a realistically shaded scene. In a first render pass the scene is drawn from the light’s position. Only the z-test is performed, resulting in a depth-map of the scene. In a second render pass the scene is drawn to screen this time from the cameras point-of-view. For each pixel the 3D position is converted into light-space coordinates and its distance to the light is compared to the corresponding value in the depth map. For most pixels this distance will be equal unless an object blocks the line of sight from the current pixel to the light. In these cases the pixel is in the shadow of another object. Additionally we smooth the edges of shadowed areas using the percentage-closer filtering (PCF) algorithm [RSC87].

Generating a shadow of the user’s hands could potentially require additional sensing and illumination. However, we are already imaging the hands of the user from the second tabletop camera used for gesture recognition and depth estimation, and there are several viable options to create realistic renderings of hand shadows in the 3D scene from this camera image.

A “naïve” solution would be to render the raw, binarized camera image onto the ground plane of the scene or as overlay on top of the entire scene – a technique often used in tabletop remote collaboration setups (see [Kir07] for an overview). Our aim however is to heighten the user’s perception that they are interacting directly in the 3D scene. With this in mind, a more

9.3 Natural Interactions Beyond the Surface 137

(a) (b)

Figure 9.6: (a) Virtual shadow cast by the users hand. Hands geometrical correctly shadow virtual objects underneath them. (b) Virtual shadows are aligned with real hand.

elegant solution involves computing the 3D geometry of the user’s hand based on the height values calculated from pixel brightness in the raw image and introducing this 3D mesh into the scene. A shadow mapping technique could then be used to generate shadows for both the user’s hand and other virtual objects in the scene. In practice however this mesh is difficult to generate using diffuse illumination alone. The main issue here is that pixel intensity fall-off is relatively accurate when hands move mostly along the Z-axis but the brightness of the captured image is not uniform throughout the space above the table. This problem is mitigated in our depth estimation by averaging pixel intensity over a region around the detected holes. In order to create a detailed 3D mesh this solution would not be viable because accurate depth information would be necessary on a per-pixel level.

An alternative approach is to first generate a shadow map for all virtual objects in the scene and then fuse this with the raw image from the camera. To do this we leverage the pixel intensity to compute an estimated Z-value for each pixel. Transforming this position into light-space coordinates produces a depth-map of the users hand as seen by the light. This depth-map can then be merged with the shadow-map by comparing the Z-value for each pixel. The larger Z- value is stored in the final shadow-map. Figures 9.6, 9.7 illustrate the hand-shadows generated by this approach.

Why Shadows for 3D Tabletop Interaction? Researchers have explored the use of shad-

ows to support a number of interactive systems. In [HZR+92] the notion of shadows were used as depth cues and as a means to open up additional DOFs for 3D manipulations on the desk- top. Shoemaker et al. [STB07] present the idea of real shadows as a mechanism for reaching across large displays. Shadows have perhaps been most extensively used for remote collabo- ration, where renderings of hands and arms of remote participants act as additional feedback mechanism for remote awareness (for an excellent literature overview see [Kir07]).

The use of shadows for 3D tabletop interaction has yet to be fully explored however, and it presents many compelling aspects. Perhaps most importantly, it gives users a natural feedback mechanism for representing their hands in the virtual scene (see Figure9.7).

138 9. Interactions in the Air - Adding More Depth to Interactive Tabletops

(a)

(b)

Figure 9.7: Combining depth sensing, pinch and virtual hand shadows to place an object into a virtual container. (a) Sequence as seen by the user. (b) Screenshots of the same interaction.

We are often unaware of our shadows when interacting in the real-world, and so they offer a subtle, non-intrusive form of feedback. However, the feedback can also be rich. For instance, how the hand shadows end up being cast in the scene and in relation to other virtual objects and shadows gives users additional depth cues allowing them to get a better sense of the 3D nature of the scene they are interacting in. For example, a user knows their hand is over a virtual object if the shadow is cast onto the top of it (Figure9.6a) whereas if the object occludes the shadow the hand is clearly underneath.