A final question is to which extent and how easy prior applications can be integrated into existing information ecologies. Prior applications support only a fraction of the document types that are used in typical information ecologies. These are either PDF documents, PowerPoint slides, Web pages or specific physical books. In contrast, our approach provides for linking not only PDF documents in their printed or in digital versions, but supports also PowerPoint slides, Web pages and all physical books which contain a EAN barcode.
Interaction techniques for annotating, linking and tagging should produce only low extraneous cognitive load. This is particularly important in situations which imply by their very nature a high degree of intrinsic cognitive load [CS91], such as univer- sity lectures. The interactions for annotation and notetaking fulfill this requirement if the underlying technology does not require the user to manually indicate page changes. The same is true for the concept of deictic gestures for following hyper- links. However, the perceptual discoupling of printed and digital media, which we have identified above, leads to a higher degree of cognitive load than our ap- proach, as users have to switch between different interaction devices. PapierCraft, the most significant previous work on paper-based tagging, requires to memorize rather complex pen gestures, which are not intuitive. These gestures may produce low cognitive load for power users. However, novices and users that utilize the sys- tem less frequently are very likely to be faced with high cognitive load. In contrast, our tagging techniques are more intuitive to use.
Moreover, the techniques should be adaptable by the end-user to make them fit the requirements of a given information ecology. Prior work does not allow the end- user to adapt the layout of the printed user interface. Moreover, the traditional practice of writing with a pen on paper is characterized by its extreme flexibility. The approach of this thesis maintains this flexibility. The user can continue making all informal comments, references, tags as before. Only if she desires that these contents are interpreted by the system, she performs a more formalized interaction, which however can be performed easily and quickly. We will detail on this approach in the next two chapters.
3.4 Discussion and Conclusions 85
the user. All systems discussed above focus on individual documents. The notes, annotations, hyperlinks and tags of users are displayed directly within the docu- ments. However, no higher-level view is given that focuses on the relations that constitute the information ecology. We present an ecological view that fulfills this requirement.
In this chapter, we have reviewed the state of the art and identified open research questions. We will address these questions in the following chapters. In Chapter 4, we will introduce a theoretical interaction model of Pen-and-Paper User Interfaces that addresses the limitations of currently existing models. Next, in Chapter 5, we will present novel interaction techniques and visualizations and an integrated sys- tem framework for paper-based knowledge work. These are based on the theoreti- cal interaction model.
4
Interaction Model of Pen-and-Paper
User Interfaces
In the two previous chapters, we analyzed how knowledge workers interact with documents and we reviewed the state of the art of paper-based user interfaces. We have identified significant shortcomings in existing models of Pen-and-Paper User Interfaces (PPUIs). In this chapter, we therefore contribute a theoretical interaction model of PPUIs.
The remainder of this chapter is structured as follows. We will first explain the concept of Pen-and-Paper User Interfaces in Section 4.1. We will then discuss three key dimensions of interaction in Pen-and-Paper User Interfaces. In Section 4.2, we present a model of the actual interactions. Based on an analytical separation of a semantic and a syntactic level of interaction, we identify interaction primitives that act as building blocks for performing semantic activities with a digital pen and pa- per. The model is of analytical value and provides support for the design of paper- based interfaces. The second dimension of the model concerns the information that is distributed across printed and digital media. We define document types, intro- duce two principles of how physical and digital information is related and provide a taxonomy of how printed and digital representations should be chosen to best complement each other. Finally, we model how users collaborate in a paper-based environment. This covers co-located and remote collaboration as well as privacy aspects. This analytical model was developed in an inductive empirical process and is grounded on our field work and on an analysis of existing user interfaces from related work. The model is the foundation of the interaction design of CoScribe, which will be presented in the next chapter.
It is helpful to define key terms before presenting the model. Our definition of an interaction model is based on the definition of [BL00]:
Definition 5(Interaction Model). An interaction model is a set of principles, rules and properties that guide the design of an interface.
Examples of interaction models are the WIMP model and Direct Manipulation [Shn83]. In contrast to implementation models, such as the Model-View-Controller concept [KP88], interaction models guide the design of the interface and of interac- tion techniques and not their implementation.
A widely accepted definition of interaction techniques is the following:
Definition 6 (Interaction Technique [FvDFH90]). An interaction technique is a way of using a physical input/output device to perform a generic task in a human-computer dialogue.
Definition 7(Paper-based Knowledge Work). Paper-based knowledge work is charac- terized by the concurrent or interleaving work with information contained on printed documents (such as books, articles, printouts of digital documents, handwritten notes) and with digital information on a computer.
Today, paper-based knowledge work is still the most prevalent form of knowl- edge work. On the one hand, knowledge work which does not include the use of computers hardly exists any more. On the other hand, as shown in Chapter 2, pa- perless offices are far from reality, as paper offers unique affordances which are not provided by digital tools.
4.1 Pen-and-Paper User Interfaces (PPUIs)
Post-WIMP user interfaces. One way to overcome the rupture between the paper world and the digital world are Pen-and-Paper User Interfaces (PPUI). PPUIs are part of the larger class of user interfaces which is called post-desktop or post-WIMP user interfaces. These interfaces go beyond the desktop metaphor and diverge from the “window, icon, menu, pointing device” (WIMP) paradigm of classical Graphi- cal User Interfaces. Van Dam defines post-WIMP interfaces as “containing at least one interaction technique not dependent on classical 2D widgets such as menus and icons” [vD97]. These interfaces build “on users’ pre-existing knowledge of the everyday, non-digital world to a much greater extent than before” [JGH+08].
Many post-WIMP interfaces extend interaction with computers into the physical space. Tangible User Interfaces, to which Pen-and-Paper User Interfaces belong, enable users to manipulate digital data by manipulating physical objects. In many cases, this results in a more direct and more natural interaction style. For example, physical cubes on a table might represent buildings. Moving a cube to another po- sition automatically adapts the building’s position in the underlying digital CAD model. In mobile and ubiquitous computing, the interaction is not bound to a fix location. Users can freely move and interact at many places. Often, the user’s lo- cation in the physical space is taken into account and influences interaction and computation. Moreover, everyday physical objects, such as fridges, cups or chairs, are augmented by digital functionality.
4.1 Pen-and-Paper User Interfaces 89
Input Optional
digital feedback
Backend system
Figure 4.1: Basic setup of a Pen-and-Paper User Interface from a user perspective
Basic setup of PPUIs. PPUIs extend computing into the physical world by turning traditional paper into a digital interactive medium. As defined on page 43, Pen- and-Paper User Interfaces (PPUIs) rely on a digital pen that leaves visible ink traces on real paper sheets and moreover digitally tracks its movements on paper. This enables transferring digital handwritings and drawings to a digital system. More- over, user interface elements can be printed onto paper which the user interacts with. For example, paper sheets may contain printed interface elements such as checkboxes, buttons, menus or fields for entering handwritten data or for issuing commands by drawing specific symbols. By interacting with a pen on these printed user interfaces, the user can control a digital system.
The basic setup of a PPUI is depicted in Figure 4.1. We distinguish two channels indicated by the arrows.
Definition 8(Channel). A channel is realized by a physical device and transfers data from the user to the digital system (input channel) or vice versa (output channel).
In this basic setup, digital pen and paper serve as a pure input channel to feed data into the digital system. Optionally, the system provides feedback via a separate channel. This is typically realized by a nearby display of a computer, a mobile phone or a PDA, but the PPUI might also use auditory or tactile output channels. Hence, the PPUI may span several devices.
In the following chapters, we will see that CoScribe has a more complex PPUI. This enables several users to collaborate with multiple printed documents using a digital pen and printed tools. Updated versions of documents can be printed in order to support system feedback on paper. Moreover, the same digital pen as on
Personal documents Shared documents Personal documents
Figure 4.2: The extended setting of a Pen-and-Paper User Interfaces of our concept
paper is used to interact with digital documents on displays. This extended setup is illustrated in Fig. 4.2.