The second goal is to tell the users how they can interact, i.e. what inter- action options are available at a specific point in time (affordances/feed- forward), how the users can perform the interactions (affordances), and what effect they would trigger (feedforward) in the application. Again, for the avatar control, there is usually no additional mechanism needed as the users do not need to perform any special actions, but even novice users soon discover that the avatar mirrors their movements. This is is similar for other continuous interactions as controlling a cursor with the hand (cf. the freehand cursor control of Section 5.3.1). It is usually sufficient to dis- play the movable objects and optionally show a hint that the users should hold their hand in front to start controlling, but the interaction itself should be straight-forward.
As soon as there is more abstraction in the interaction, more information on how to interact is required. One example is the more abstract cursor control of the “swipe keyboard” described in Section 5.3.1. In the “swipe keyboard”, the vertical movements of one hand are used to switch between different groups of onscreen items, while the vertical movements of the other hand switch between the elements of that group. A horizontal movement of the second hand then actually selects the currently targeted GUI item. As the interaction is divided on both hands, it is quite different to classical GUI interaction, in which only a single cursor is controlled with one hand. Therefore, the users need an introduction before they can start interacting. Nevertheless, in the mentioned case, the information on how to interact was provided by the experimenter in the introduction of the study. For
an actual product, it would be needed to embed this information in the system itself. Freehand GUI interaction includes other parts that require more helping mechanisms, e.g. highlighting and/or playing a sound when the cursor moves over an object that one can interact with. An example for affordances in freehand GUIs would be information on how to perform the actual selection gesture. Nevertheless, in the case of freehand GUI interaction, this gesture should be very short and easy to remember, so that it should again be sufficient to provide this information only at the beginning of the interaction.
With more complex body gestures, affordances and feedforward get more important, and this is also the part in which FUBI offers more mechanisms to support the interaction designer in providing this information. Those mechanisms are mainly included in the two game engine integrations of the FUBI framework. The major way to provide affordances and feedforward in FUBI is to link the input gesture with a (labeled) gesture symbol (static image or animation) as illustrated in Sections 3.1.2, and 7.4. In Section
3.2.1 an animated gesture symbol was even automatically generated out
of the gesture’s XML definition. At any point in time, the possible input gestures are displayed on the screen using the linked symbols (affordances), optionally enhanced with text labels that describe the corresponding sys- tem actions (feedforward). The action labels are usually not displayed if they are already represented precisely by the corresponding gestures them- selves (combined affordances and feedforward), or the application is de- signed to hide that concrete information (hidden feedforward). Of course, those mechanisms can also be omitted in the case of frequently required interactions which are learned after a while, as the gestures used for navi- gation in Section 3.2. If there are too many interaction options to display them all on the screen at the same time, the symbols can be organized in a hierarchical structure as well, e.g. similar to the swipe menu in Section
7.4.2 that needs to be triggered with a specific “talk” gesture.
The automatic generation of animated gesture symbols based on virtual characters as introduced in Section 3.2.1 was further enhanced at a later stage. Generating the gesture symbols out of the Fubi gesture XML has
(a) Virtual Character left: textured;
right: depth colorized
(b) Active Limb Highlighting
(c) Corrective Arrows
Figure 6.1: Gesture Visualizations with a Virtual Character
the advantage that the symbols do not have to be created manually, they can still be easily adapted and it is also possible to add further enhance- ments, e.g. for additional affordances and feedback/-forward information or a specific shading technique. The generated gesture symbol already used a shading technique that encodes depth information with colors, which is now improved to apply colorization from magenta = near to red = far (see
Figure 6.1ain the right-hand image). This reasoning behind this is that one
major problem for the comprehensibility of the symbols can be the missing depth perception.
As the automatic generation in Section 3.2.1 only supported very few gesture types, it was first enhanced to support all kinds of gestures that can be defined in FUBI gesture XML. In the meantime, there also multiple additional options to include information into the gesture symbol, e.g. we highlight limbs which are involved in the current gesture (see Figure 6.1b) or add arrows depicting how to move the corresponding joint for correctly performing the gesture (see Figure 6.1c). To make the generation easier to configure, optional parameters can be configured depending on the gesture type, e.g. one can define a pre-stroke posture, additional scaling, concrete timings for basic gestures, or a general tolerance value.