Based on our findings, we propose a set of design improvements for our techniques, which are detailed here. We moreover improve our system prototype and implemen- tation described at the end of this section.
3.5.2.1 Halo as a Group Visualization
We improved our design so that once occlusion takes place, our system visualizes an interactive halo representation to provide awareness of occluded items. The
3 Replacing lightweight physical occluders was easier than dragging out access. 3 For complex occluders, drag out strategy was mostly used to resolve occlusion. 3 Proxies were found to be helpful particularly for accessing fully occluded items. 3 Icon representation turned out to the most suitable level of awareness.
3 PressView technique was particularly helpful for a quick peek into occluded items.
7 Miniature was found to be distracting and introduced too much visual clutter. 7 A group accessing technique was missing.
7 Functional zones was found to be suboptimal for indicating the default aware- ness level.
Table 3.5: Summary of results from the exploratory experiment
halo is visualized around the occluding physical object as illustrated in figure 3.22
regardless of the number of occluded items. The occluded object remains in its original location (R1) but an icon-sized miniature representation is visualized on the halo. This indicates the rough spatial location of the occluded object relative to the occluding physical object and provides additional information about the occluded object.
Figure 3.22: Halo visualization and iconized representation of the occluded objects around it
to avoid permanent clutter, upon releasing the fingers, objects that were not moved snap back to their original, occluded locations.
The halo visualization is inspired by the non-interactive Glow technique intro- duced by Javed et al. [Javed 2011]. In contrast, both our halo and the individual proxy designs act as interactive widgets to preview and access occluded objects. Icons provide access to individual occluded objects (R1). In order to access all un- derlying objects users can also drag out and enlarge the halo (cf. figure 3.23c,d). The halo visualization enables global awareness of occluded objects as well as access to occlusion groups and individual objects, while minimizing visual clutter.
3.5.2.2 Remote Accessing
Based on participants’ feedback, we improved the remote zooming gesture and adapt it to perform in combination with the halo visualization. While touching a halo or an individual icon with one hand, the other hand performs a pinch gesture on any other area of tabletop display as depicted in figure 3.24. The pinch gesture opens an interactive preview of the object or the occlusion group, offering the same functionality as the direct technique described above. By pinching the fingers apart, the preview gets larger and shows more detailed information. Once the preview is opened, the user can release the finger on the icon to interact with the preview or to reference to another halo. The preview disappears once the pinch gesture is released. Note that with the improved remote access gesture users not only can temporary inspect an occluded object but access it on any empty area on the tabletop. Moreover the refined version enables a remote group access (cf. figure 3.24b), which was not possible with the previous remote zooming gesture.
3.5.2.3 System Prototype, Improved
We improved our system prototype by using the state-of-the-art tabletop device. We reimplemented our prototype, which runs on a Samsung SUR40 with an active
(a) (b)
(c) (d)
(e) (f)
Figure 3.23: Accessing occluded objects at different scopes: (a and b) individually, (c and d) group access, and (e and f) showing the global expose technique for getting a global overview
(a) (b)
Figure 3.24: Remote accessing technique – users can select either (a) an individual proxy or the halo for an individual proxy or (b) a group access, accordingly.
display area of 40 inches and resolution of 1920 x 1080 pixels. The system also includes a computer with a dual-core processor with 2.9 GHz. In contrast to the previous system which utilizes a camera for touch tracking and marker recognition, it uses the Microsoft PixelSense technology4 for touch and physical object recognition on its surface without the use of cameras.
The PixelSense technology (cf. figure 3.25b) basically uses a back diffuse il- lumination approach, which with the aim of a high-resolution grid of integrated sensors can beam as well as observe IR lights reflecting back from the fingers and objects footprints. As it can be seen in figure 3.25, the SUR40 offers a relatively compact (only four inches thick) and more table-like form factor so that users can conveniently sit around it. This feature makes SUR40 particularly suitable for user studies.
The object recognition on the Samsung SUR40 system is realized through fiducial tag markers. Thanks to the Microsoft PixelSense technology, the tag markers can be seen and decoded with the size of only 19 x 19mm. The marker basic coding scheme is composed of four reference points – namely, one large point in the middle of the marker for extracting the position and three guide points to specify the orientation of the tag. In addition, each tag contains from zero to eight data bits that define the tag value. These data bits are white circles (0.075 inch radius) that are clockwise centered around the tag.
We reimplemented the occlusion-aware techniques described above in a fully functional system written under .NET framework in C# programming language. Tag markers enable tracking physical objects on the tabletop surface. Although
4
(a) PixelSense Technology
(b) Samsung SUR40
Figure 3.25: Samsung SUR40 tabletop system which uses Microsoft Pixelsense tech- nology
to either (left, top, bottom) or (right, top, bottom) as possible placements.
In the next step, the distance to the top edge of the physical object is compared to the one to the left border, if it is smaller, then the object gets visualized at the top border. if not, the same is done for the bottom edge. If neither of them was closer, it is finally visualized on the right. For placement at top or bottom edge, the original x-coordinates are retained and y is set so that the icon is above or below the object; for left and right edge placement, the same is done but with y value retained and x value set accordingly.
The software also recognizes two-handed organizational gestures. Basically, the touch points is grouped into gesture objects based on their proximity when they first appear. The gesture object includes: number of fingers in the gesture, speed, duration, acceleration, angle, size, plus a state that can be set to active (i.e., touch points still present) or inactive (i.e., gesture ended, this state is seen once by all subscribed objects and then the gesture object is freed).
Then, a visualizer component employs a simple decision tree to determine the gesture means. For instance for teleport, if the visualizer sees an inactive (termi- nated) one finger gesture with a move distance of at least x mm and a duration less than y ms, then it is considered a swipe. The visualizer then asks the gesture processor if there is any active three finger gesture on the table; if so, the object gets moved there. Another example is rotating: if the visualizer has an active two finger gesture on it, it adds the rotation delta of this gesture object to its own rotation value. In this fashion, all gestures are detected independently and perform more robustly.