La administración

Several other promising applications domains exist for PPI and mPPI. Pen based input naturally supports free-form inking, drawing and writing. Thus, several applications support design related activities, e.g., sketching, through PPI. For similar reasons, a section of PPI based applications support applications in therapy, e.g., for elderly people, as well as collaboration and interaction on large scale, or ”difficult to control” displays. Furthermore, the easy pointing and navigation on paper surfaces prompts several applications requiring a convenient remote control functionality to employ PPI. Finally, several applications supporting command and control scenarios leverage the combination of physical affordances of paper and digital systems through PPI.

Creativity

In a structured sketching approach, Dachselt et al. extended a UML modeling tool by PPI enabling intuitive input stimulating creativity within the formal visual alphabet of UML [Dachselt et al., 2008]. Thereby, the combination of pen and touch offers a novel palette of interaction techniques supporting collaborative activity in structured diagram design [Frisch et al., 2010]. Digital pen technology also naturally supports collaborative sketching in co-located sessions [Geyer et al., 2012]. Holzmann and Vogler extended this notion in a prototyping system: users can sketch user interfaces for mobile devices generating a paper prototype that can be converted to an executable prototype on the mobile device [Holzmann and Vogler, 2012].

Modelcraft, [Song et al., 2006, Song et al., 2009b], extends sketch based input into 3D space, by introducing a set of PPI based gestures for manipulating the digital rep- resentations of 3D paper models. This allows supporting creative techniques for ar-

chitects during early model building phases. Catch-Up 360, [Perteneder et al., 2015], follows a similar approach and facilitates remote collaboration among industrial de- signers by enabling PPI based input on the surface of realworld, three dimensional models. This setup also adds projection based feedback to editing operation carried out with digital pens.

Toward supporting other groups of specialists, Tsandilas et al. demonstrated in

Musink, [Tsandilas et al., 2009], how visual language drawing techniques can be used to support music composition via PPI. In their approach, the composer can write notes onto paper. Subsequently, notes will be transformed into playable midi se- quences in the digital system. PaperComposer, [Garcia et al., 2014a], bases on the same concepts, yet adds a full-blown specialized PPUI builder for musical user in- terfaces, i.e., interactive regions triggering playback of audio files. PaperTonnetz, [Garcia et al., 2013], provides an application employing such a musical PPUI. It of- fers musicians the interactive exploration of two dimensional structures on paper, that correspond to audio sequences, i.e., playback of a Tonnetz24.

However, these approaches exclusively target stationary settings for drawing, de- spite creative activities often involving the factor mobility, e.g., in creative discussions outside the normal workplace.

Mobile Creativity. Toward mPPI, Tsandilas introduced a mobile system extend-

ing Musink to help users interpret digital ink during creation using a combination of digital pen and touch input on smartphones [Tsandilas, 2012]. The system strikingly demonstrates the potential of hybrid mPPI ensembles: as in this particular task cor- rect recognition of digital ink becomes crucial, it allows users immediately reviewing recognition quality (and correcting potential misinterpretations) of digital ink. This heavily bases on supporting user mobility and employing the interactive operation mode of the digital pen. Furthermore, digital pen technology offers the creative free- dom and flexibility required in music composition tasks which cannot be supported by traditional, GUI based systems alone, [Garcia et al., 2014b].

PaperCAD, [Lee and Stahovich, 2014], represents another interesting step toward mobile creative support demonstrating the potential of mPPI and hybrid mPPI en- sembles. It enables engineers to interactively explore computer aided design (CAD) drawings in mobile settings. As such it provides acoustic information and videos asso- ciated with certain interactive regions overlaying CAD documents. However, its PDA based prototype lacks supporting infrastructure that would allow other applications to share and access its resources, i.e., falls short of supporting document mobility.

UbiSketch, [Cowan et al., 2011, Weibel et al., 2010a], is a mPPI based application for communication in social networks using small sketches. The system continuously

streams digital ink of user drawings on paper to a smartphone. Subsequently, the digitized drawings can be published on a social network site as images. It supports user mobility and the interactive operation mode in a limited way, yet neglects other mobile aspects, e.g., document mobility.

Recently, Ha, Park and Lee introduced a low-fidelity prototyping application based on mPPI, [Ha et al., 2014]. It enables designers to draw prototypes of mobile appli- cations and directly transform these paper prototypes into executable models using mPPI. Their applications demonstrates the potential of hybrid mPPI ensembles in the application development process: paper prototypes become executable directly on tar- geted mobile devices.

Therapy

Other applications of PPI focused on user groups with special needs. Piper et al. presented an application in speech therapy, where users suffering form aphasia and apraxia can train their speech abilities with the help of the LiveScribe Pulse Smartpen [Piper et al., 2010, Piper et al., 2011]. Similarly, Memento, [West et al., 2007], aims to support elderly people in episode recall. It provides a multimedia scrapbook appli- cation on a desktop PC, where multimedia content (photo, video, audio) can easily be annotated and managed using PPI.

Piper, Weibel and Hollan found in a long term study investigating a similar ap- proach that PPI offers a huge gain for social interaction and communication at an advanced age [Piper et al., 2013]. The authors subsequently introduced the design of a PPI based application assisting in communication therapy for elderly people, [Piper et al., 2014]. This also presents the first step towards mPPI: the LiveScribe platform used in their prototype allows for mobile use although limited to user mobil- ity, i.e., document mobility is not supported.

Recently, Prange et al. successfully applied PPI in an application assisting demen- tia patients in social communication and preservative training of cognitive abilities, [Prange et al., 2015]. In their approach, patients control robot companions through a mixture of natural language and PPI. Thereby, the robot companions provide assis- tance in certain cognitive tasks, e.g., reminder functions.

Collaboration

Large paper surfaces can support co-located collaboration, e.g., for safety critical ap- plications in a military setting [Cohen and McGee, 2004] or in collaborative knowl- edge work [Nyu and Miura, 2011]. An early application of this concept is the pro- totype introduced by McGee et al. [McGee et al., 2002] (c.f., command and control

applications below): Here users can orchestrate missions using a paper map as central discussion and planing board and PPI to execute command and control tasks.

GigaPixel Prints, [Yeh et al., 2006b], further explores this idea through several small prototypes. In this approach, large sheets of paper serve as displays with a high spatial resolution in combination with a low temporal resolution (re-printing). This concept has been extended by Haller et al. in their augmented meeting room [Haller et al., 2010]. Their approach enables interaction with several digital systems deployed in a meeting room including large print-outs, tabletop PCs and wall sized interactive surfaces through PPI in combination with Anoto enabled surfaces. This concept supports, e.g., collaborative product design activities employing the large screen real-estate of the augmented meeting room and the creative freedom of PPI [Geyer and Reiterer, 2010].

In addition to support during meetings and collaborative activities, PPI lends itself to document and share meeting minutes [Ispas et al., 2010a]. Furthermore, Weibel et al. explored PPI based support for collaboration over distance using a remote sketch- ing tool [Weibel et al., 2011b]. Users can draw on individual paper sheets using digital pens, while a shared virtual drawing board visualizes the drawings of all collaborators. Collaboration is further facilitated by using Skype25 as voice-over-ip environment, where users can discuss their ideas while drawing.

However, these approaches exclusively target stationary settings, mPPI based sup- port for mobile collaborative practices remains unexplored.

Paper as Remote Control

Signer and Norrie demonstrated that linking physical regions on paper to digital func- tionality facilitates PPI based remote control systems, e.g., to control and draw on slides in a presentation [Signer and Norrie, 2007b]. This concept has also been used in the commercial Oxford Papershow product26. In a similar approach, the program guide leaflet allows controlling the television program [Berglund et al., 2006]. Here the user manipulates the television program by check-marking interesting regions in the leaflet.

pRemote, [Hess et al., 2008], extends the notion of a paper based remote control to a universal personalized remote control system. Here, the user configures digi- tal functions needed by drawing the layout of the remote. Thereby, the system at- taches digital functionality to drawings using an end-user service composition ap- proach [Borggr¨afe et al., 2008]. This paper based approach to end-user programing increases user satisfaction [Hess et al., 2011].

25_{http://www.skype.com (accessed: July 2015)}

Weibel et al. suggest to use such a PPI based remote control approach to in- teract with large scale displays: as the user cannot easily reach many parts of the screen, configuration takes place on a paper placeholder card in the Hiperpaper sys- tem, [Weibel et al., 2010c, Weibel et al., 2010b]. In a similar fashion, Vodoosketch, [Block et al., 2008], explored a tool palette printed on paper to conveniently interact with large vertical surfaces, e.g., drawing boards.

Despite remote controls offering an interesting mobile use case, these approaches target only stationary settings.

Command and Control Centers

PPI also serves as an important modality in command and control centers. Appli- cations thereby range from air traffic control [Vinot et al., 2014] over military appli- cations [McGee et al., 2002] to collaborative table top systems for first responders in emergency situations [Doeweling et al., 2013]. In this context, Cohen and McGee in- troduced Rasa, [Cohen and McGee, 2004], a prototypical system supporting military officers in the field by offering multimodal input on paper artifacts. This enables them to orchestrate military operations while using the convenient tangible properties of paper artifacts.

Letondal et al. followed a similar approach in Strip’TIC, [Letondal et al., 2013], a system designed to support air traffic control using the physical affordances and in- herent support for co- located collaboration of paper artifacts. Here, however, paper artifacts are overlaid by digital information via projection and interactivity is increased by means of gaze tracking [Hurter et al., 2012]. D¨oweling et al. applied this approach to the domain of emergency first responders, where a wealth of heterogeneous in- formation and differing media, e.g., paper notes and digital artifacts such as emails, had to be integrated in an intuitive and robust way to support interaction under stress [Doeweling et al., 2013].

Discussion

The applications discussed in this section demonstrate a plethora of additional appli- cation domains for PPI and mPPI. PPI naturally and conveniently supports creative tasks and co-located collaboration, as well as applications in therapy and interaction with large surfaces. However, current systems in these domains also mainly focus on supporting stationary settings. Only the domain of creative tasks poses a notable exception: examples of creative applications, e.g., the mobile music composition as- sistant, [Tsandilas, 2012], strikingly demonstrate the potential of hybrid mPPI ensem- bles. However, these approaches lack insights toward design of enabling infrastruc- tures, conceptual frameworks and interaction theory.

2.3 Infrastructures for Mobile Pen-and-Paper