Several prototype applications were developed in order to demonstrate that the dis- tributed interaction processing pipeline allows supporting mobile PPI. Besides demon- strating feasability, they also aim to show practical relevance of the approach and outline how different deployment schemes contribute to the overall flexibility of the approach in various domains. This serves as the second part of the proof-of-concept evaluation (the reference implementation Letras constitutes the first part, c.f., section 3.3).
Mobile Applications
Mobile application prototypes form the central element of the proof-of-concept eval- uation as they show how mPPI infrastructure based on the distributed PPI processing pipeline actually enables mobile use of PPI. In order to evaluate this claim of en-
Figure 3.13: Prototype applications: a. the magic drawing application, b. cross-media bookmark application, c. the digital / physical task list
the same processing pipeline which bases on the adaptions for mobile platform usage as described in section 3.3.2.
Digital / Physical Task List This application combines the communication facili- ties of a mobile phone with traditional paper as input medium in a todo appli- cation, or task list. It is designed to allow noting tasks on any (Anoto enabled) paper document. As shown in Fig. 3.13 (c.), this can be achieved by simply drawing two opposite corners on any paper artifact where the user wants to note down a task. Handwritten tasks can then be shared with and delegated to other people. It is also possible to interact with tasks, e.g., complete, delete or edit them, using either mPPI or interaction on the smartphone. Thereby mPPI functionality enables users to mark a task as done using a simple check mark gesture, cross out a task in order to delete it, or edit the task on paper.
This application’s implementation predates the MobiLetras mobile platform. Although it also hosts the driver and region stages directly on the mobile plat- form, no parts of the semantic and application stages are used. Gesture recog- nition and digital ink storage are implemented directly within the application. Thereby, gesture recognition bases on a simple implementation of the $1 Ges- ture recognizer [Wobbrock et al., 2007]. This application demonstrates how the distributed processing pipeline allows creating and disseminating new interac- tive regions (here: paper areas where the user writes down tasks) on the fly, i.e., during runtime. This exemplifies its power regarding resource sharing, where not only the digital pens, but also the paper are itself can be shared between applications and demonstrates a scenario requiring document mobility support at the infrastructural level.
Magic Drawing Application This application allows the user drawing on a sheet of paper (or write notes and other information), while the digital ink is recorded by the smartphone. Notes are represented using a digital ink facsimile on a digital representation of the paper sheet in the user’s smartphone. The user can instantly interact with this digital version using the phone itself as an interac- tion device. She can change to various pencil stroke widths and colors, however, those will only be visible in the digital facsimile. The paper version is consid- ered only a draft. In this application, Pidget interactors on special pre-printed paper documents allow using commands directly on paper as shown in Fig. 3.13 (a.). It is possible to flush away and store the digital facsimile by shaking the mobile phone. This simple application derives its name from the magic drawing table kids use, where sliding over the drawn contents erases them instantly. The Magic Drawing application uses the MobiLetras infrastructure in combina- tion with the interaction toolkit based on the conceptual framework for PPI in- troduced in chapter 4. It interprets digital ink stemming from interaction occur- ring on its interactive regions within the semantic stage of the pipeline (through the recognizers encompassed in the toolkit, c.f., section 4.3.3). Thereby, inter- active regions are the specially prepared paper documents, as well as the Pidget interactors printed on them. This applications demonstrates the utility of the mPPI infrastructure with respect to user mobility: it employs the interactive mode of the pen, bases on the mobile platform and uses a specialized deploy- ment scheme to adapt to this setting.
Hybrid Photo Scrapbook The hybrid photo scrapbook was modeled after appli- cations described in the literature, i.e., ButterflyNet, [Yeh et al., 2006a], Me-
mento, [West et al., 2007], and Prism, [Tabard et al., 2008]. It enables users to take notes on paper and enrich those with digital photos captured using a smart- phone. Users can capture their notes using the digital pen on paper, while a facsimile of the digital ink is stored on the mobile device. In addition to digi- tal ink, they can insert photos into their notes by drawing placeholders on the paper. This application also allows triggering the mobile device’s camera di- rectly when the user draws the placeholders to demonstrate the increased level of interactivity through supporting the interactive mode during mobile use. Similarly to the magic drawing application, this application bases on MobiLe-
trasand the interaction toolkit. Gesture recognition uses the $1 Gesture recog- nizer [Wobbrock et al., 2007] integrated into the pipelines semantic stage, stor- age of digital ink bases on a shared SQLite based data store being part of the application stage. It demonstrates that in this setting, infrastructural support for user mobility enables convenient use of mPPI.
Cross-media Bookmarks The cross-media bookmark application was modeled partly after CoScribe, [Steimle, 2009a]. It demonstrates in a limited prototype, how the distributed PPI processing pipeline enables a stationary application to be used in the mobile domain. As shown in Fig. 3.13 (b.), it enables users to take notes and link those notes with digital documents using cross-media links. Thereby, the user can mark certain regions of physical documents and thus con- nect these documents with web pages. Whenever the mark is subsequently touched with the digital pen, that particular web page opens on the smartphone’s browser. Several interaction techniques exist to establish the bookmarks and will be introduced in detail in chapter 5.
As the magic drawing application and the hybrid photo scrapbook, this applica- tion bases on MobiLetras and the interaction toolkit. Gesture recognition uses a custom implementation for bookmarking gestures. Whenever bookmarks are established, an interactive region around the mark is created to serve as Pidget interactor for accessing a particular web page.
Integrated Note-taking Application This application combines digital and phys- ical note-taking applications into a cohesive system. It enables the user to take notes on arbitrary (Anoto enabled) paper artifacts while being on the move and to integrate a digital facsimile of these notes with a digital note-taking system, i.e., the Evernote cloud based note-taking application20. Notes can be taken on any paper artifact by drawing two opposing corners of the note’s area, a ges- ture similar to the one used in the digital / physical task list or in the hybrid scrapbook. Notes can later be simply re-accessed, i.e., their digital facsimile displayed on the mobile device, by tipping the pen on them. Additionally, the user can control the stroke width and color of generated digital ink through the mobile device as in the magic drawing application. Inserting photos into notes as in the hybrid scrapbook is also supported. As such, this application can be seen as a comprehensive successor to the aforementioned prototypes, integrat- ing their capabilities with the Evernote system.
This application bases on MobiLetras where both driver and region stage are deployed on the mobile device itself. It also uses an SQLite based storage in the application stage to store digital ink in its original form on the mobile device, as well as a remote store where it synchronizes notes with the user’s Evernote account. This makes their facsimile accessible on other devices, e.g., the user’s desktop computer and demonstrates a slightly differing deployment scheme, where two independent application stage storages are used.
Mobile SVG Editor The mobile SVG editor is another successor to the magic draw- ing application. It supports the same functionality, the drawing of diagrams or notes in mobile settings extended by the capability to export any drawings di- rectly to the Scalable Vector Graphics (SVG) format21. Attributes of the digital facsimile, e.g., pen stroke width and color, can be controlled using the mobile device and paper. In contrast to the magic drawing application, the SVG edi- tor does not require any pre-printed paper documents for applying controls on paper. The user can convert any (Anoto enabled) surface into a a drawing area by using the corner gesture. These regions, i.e., notes, can be re-accessed when tipping on them later. Digital ink properties can be controlled on the phone. Additionally, the user can draw pie-formed interaction proxies, that allow to bind a certain functionality to them. The size of the pie determines which func- tionality will be selected by a linear mapping of circumference to menu item on the mobile device. Whenever those pies are tipped, the bound functionality is invoked, e.g., the pen stroke color is changed.
This application also bases on MobiLetras. It uses a custom-built gesture recog- nizer for pie-based gestures and handles interaction proxy registration by creat- ing new interactive regions bound to certain functionality. This exemplifies the flexibility of the publishing and discovery system for interactive regions within
Letras. Additionally, a SQLite based data store is used to store drawings. Digital Grocery List (DGL) The DGL supports grocery shopping in a mobile use
case and requires both, user and document mobility support in the underly- ing infrastructure. It combines a mobile and desktop client for collaboratively editing and sharing grocery lists on paper and in the digital domain. This appli- cation will be discussed in detail throughout section 3.4.3, as it constitutes the case study reported in this section.
This set of applications shows that the distributed PPI processing pipeline enables the infrastructure to support the mobile use of PPI. It shows that several heterogeneous applications can easily be developed given that the infrastructure provides adequate support. Bringing stationary applications of PPI to the mobile domain, e.g., as with the hybrid photo scrapbook, becomes possible through to infrastructure satisfying the requirements for mobile usage.
Desktop and Smart-Environment Applications
Besides the mobile use case, PPI infrastructure also has to support stationary appli- cations or applications being executed in a smart environment. Toward this end, it is
important to use the same pipeline otherwise the advantage gained by sharing inter- action resources through the distributed processing pipeline would be lost. All in all, the same pipeline need to be able to support stationary applications also. In order to validate this and to complete the proof of concept evaluation, a set of stationary ap- plications was developed. It demonstrates that the distributed PPI processing pipeline supports stationary applications in parallel to mobile applications.
Table-Top Control The use case of this application is control of a table-top dis- play by means of a digital pen in a smart environment. In this environment, the user can use the same digital pen to interact with several other applica- tions and resources to facilitate seamless interaction. This application was ini- tially implemented as an adaption of an earlier prototype developed by Steimle, [Steimle, 2009a], to evaluate Letras capabilities as supporting infrastructure for legacy systems. In Steimle’s application the pen is used as direct pointing device on a rear-projection table-top display. This had been rebuild based on Letras. Thereby, it was possible to considerably reduce the footprint of the overall appli- cation by using Letras instead of the original infrastructure (based on an adapted
PaperToolkit, [Yeh et al., 2007], branch). The deployment scheme here hosts driver and region stages on the table-top. It does not require semantic process- ing or storage of digital ink, as this is handled within the legacy application. However, it defines an interactive region corresponding to the table-top screen and this wrapper delegates digital ink to the application itself.
Emergency Response Workflow Support As paper centric workflows still per- sist in control and enactment of emergency response processes, this application developed by D¨oweling et al., [Doeweling et al., 2013], supports the integra- tion of such workflows into the digital world by augmenting a tabletop display in order to allow for interaction with digital pens. At the same time it includes interaction with paper artifacts in a smart control room.
This applications uses a deployment scheme where pen connection and region management are hosted in the smart control room on a central server. The user can control a table-top screen by means of a digital pen, while the same digital pen can be used to interact with a variety of applications including notes and emergency plans in the surrounding environment.
Collaborative Drawing and MindMapping Support This application aims to support group discussion in an augmented meeting room by offering a big central drawing surface (A0 format) onto which users can draw and write by means of digital pens. It supports simultaneous collaboration of multiple22
users. Thereby, each user employs a pen producing different colors in the gener- ated, digital facsimile. After the group discussion, the contents of the discussion surface, e.g., contained drawings and mindmaps, can be exported to SVG and published either on a web server, or used in personal notes.
This application is deployed with region and application stage on one worksta- tion per room, while the driver stage will be hosted on a set of other nodes, e.g., to allow for more digital pens being used simultaneously. The application stage uses a Mongo DB23backed data store for persistently storing digital ink. PPI Based SVG Editor This application presents a stationary version of the SVG
editor developed as a prototypical mobile application as described above in sec- tion 3.4.2. It shares its features and enables users to interact with the system by means of pie based interactors. Users can draw these interactors on paper in order to select pen stroke width and color. Drawings can be exported to SVG. It’s gesture recognizer is a custom-built recognizer that allows calibrating and fine-tuning recognition properties (as used in the mobile version).
Deployment here is a simple flexible deployment, that only requires one version of the pipeline and its services somewhere in the vicinity24.
Besides the applications presented above, various further systems base on Letras and its associated components. Letras has also been adopted by other researchers in the context of their scientific work, e.g., Lisserman et al. used it in order to add pen interaction to their research prototype [Lissermann et al., 2012]. For the sake of brevity, this section cannot enumerate all systems basing on Letras.
Despite initially being designed for stationary contexts, the applications presented above have successfully been tested in integration with personal digital pens con- nected to a user’s personal mobile device hosting the driver stage, e.g., as part of one of the mobile prototypes. Here the digital pen resource can be shared and the same digital pen can be used, e.g., for collaborative drawing and other applications.
This shows that although all these applications were developed separately, they allow to share the same interaction resources through the Letras distributed processing pipeline. It is possible for example, for the user to easily control the graphical user interface to her shopping and task list with the same digital pen she uses to write on the collaborative drawing surface of the mindmapping application and the same pen she uses to interact with the tabletop system.
bluetooth personal area network (PAN) limitation of max. 8 devices
23http://www.mongodb.org/ (accessed: July 2015)