data
state understanding
graphical structures & marks
perceptible & manipulable state
perception transformation display transformation display transformation^-1 controls user controls transformer meaningful transformation external input
instruments & structures controls
trigger and control
Figure 5.4: Controlled Transformation
rely not only on computing (separated into two disciplines in the figure: numerical analysis and symbolic computing) but also on Interaction, through interactive algorithms. Those algorithms do not compute values: they manage the execution flow differently from the procedural paradigm, and they change the states of a program as opposed to compute values in a so-called “functional” manner (i.e. without changing the state). Any practical program needs those algorithms [SS78]. Therefore, instead of considering Informatics as the “Science of Automatic Computation”, I consider what I and my colleagues do as the “Science of (meaningful) Controlled (automatic) Transformations” (figure 5.4), summarized here as “Science of Controlled Transformations”. “Transformation” contrasts with “Computation”, considered here as a means. “Controlled” con- trasts with “Automatic”: if some of the underlying transformations are automatically done by a machine and are extremely fast, they are meaningful only because a user triggers and controls them according to its perception of their results.
One can argue that considering computing as a special case of transformation leads to a “Trans- former Science”, and that this science would not be much different from “Computer Science”. Nonetheless the device, the transformer, is only one part of the story: in contrast to the concept of computation, the concept of transformation encompasses not only the device (P&M State + transformer + unperceivable state) but also the user as we can see on figure 5.4.
5.4
My work considered as research on controlled transformations
In order to explain and demonstrate how we can consider Informatics as the Science of Controlled Transformations, I will reinterpret the work presented in the previous sections as controlled transformations.
5.4.1
Transformation of evolving data and graphics
The invisible conceptual state may change because of external input: geographical positions of vehicles at the airport, passing time, or change by teammates. All these changes can be considered as transformations. One problem for users is the visibility of these transformations. For example, a simple update consisting in replacing old state values with new ones that triggers a sudden change of representations would be unnoticeable by users either, because the chances are that the change would not occur under the users’ focus of attention. Similarly, changes that occur over a
!
Figure 5.5: Animation, MAMMI’s “vapor” trails, expanding targets and videoProbe’s aging of pictures are meaningful transformations of evolving data and graphics.
long period of time can go unnoticed because the change in the representation is very subtle. This problem is also known as “change blindness”.
My contribution to this topic is to have designed graphical transformations that help users notice and understand the changes. More precisely, together with my colleagues we designed automatic smooth transformation of graphics to facilitate the perception and understanding of uncontrolled abrupt changes of data (figure 5.5).
For example, cartoon-like animations in the Anims project are a way to make the changes, or the transformations, more noticeable and understandable. The aging of pictures in VideoProbe is a way to make the user notice and understand slow-evolving data. Similarly, the changes made by a teammate can be unnoticed by a fellow temam member in a CSCW system, as if those changes were made by external input. The “vapor” traces left by gestures on the surface in the MAMMI project can also be considered as a transformation designed to be noticeable by partners. Graphics themselves may require gradual change to be understandable and informative. The rolling-dice animation of FromDaDy is an example of such a transformation. This particular transformation is either triggered by the user and automatically performed, or triggered and controlled by the user. This enables the user to slowly perform the rotation, go backwards, and follow marks on the screen in an accurate manner.
Those transformations are meaningful as they are designed in such a way that users understand the transformations and they can grab information from them. The ecological aspect of the design (“flying” strips, aging pictures, transient vapor, rotation) makes them intuitive: users can rely on their knowledge of the real world and their hard-wired perceptual system [Gib86] to interpret them directly.
Finally, the expanding targets are a transformation of the graphics that aims at facilitating the acquisition during a pointing task. The control of the transformation is handled by both the device and the user. The transformation is controlled by the gesture of the user, with “controlled” used in the sense of “control theory” (“asservi” in French). The user does not consciously trigger the expansion: the device itself triggers the animation when 90% of the distance has been covered.
5.4.2
Action and exploratory design considered as transformation
Users are often involved in “design” activities, in a broad sense of the term: editing a spreadsheet, a drawing, a presentation etc. Even Air Traffic Control can be considered as a design activity: users plan future actions according to the upcoming traffic. The manipulations that users perform are controlled transformations from a state of the “scene” they edit to another. When controllers give instructions to pilots, they both act on the state of the traffic, and on its future state.
5.4. MY WORK CONSIDERED AS RESEARCH ON CONTROLLED TRANSFORMATIONS 47
! !
Remove KeepExtend
Shape Fill Stroke Thickness 1 Samples ManySpector
Figure 8: a) bi-directional grooved str
Figure 5.6: FromDaDy’s “brush, pick and drop” and mapping tool, ManySpector and the bidirec- tional Strip’TIC board are interactions to help perform meaningful transformations.
My contribution in this area is the design of interaction techniques to prepare and perform immediate, powerful and meaningful transformations, as well as future transformations. Figure 5.6 illustrates this aspect of the work I have done with my colleagues. The brush, pick and drop interactions of FromDaDy are controlled transformations of multiple objects at the same time to foster better perception. The FromDaDy interface to set the mapping between data and visual variables is a transformation which is defined (and thus controlled) by the user that is automatically executed by the system (in other words, the user programs the transformation function of data to graphics). ManySpector leverages the implicit structure of the scene to foster transformations of multiple objects at once, while the explicit structure with dependency links is a way for the user to prepare for and optimize future transformation of multiple objects at once. In a sense, this is a way for users to make future transformations usable. The MAMMI clearance editor helps controllers prepare and delay execution (triggering) of transformations of flight parameters in a collaborative way. The Strip’TIC mixed interface enables users to transform flight parameters using a regular pen and position tracking.
5.4.3
Transformation from graphics to mind
Perception itself can be considered as a transformation of graphics to information in the users’ mind. A “good” representation is transformed in an efficient way by our perceptual system into structures which in turns are transformed into information and insights. This transformation from graphics to mind is “automatic”: it is always there, users do not trigger it, they perform it permanently. A part of the transformation can be improved with training, but a part of it cannot be changed, it is hard-wired in our perceptual system and users cannot control it. Therefore designers must design for this “machinery”, there is no way avoiding or ignoring it.
My contribution in this area is the work I conducted in order to gather knowledge about the machinery and the hard-wired transformations (figure 5.7). The work on visualisation charac- terization, the ScanVis framework and the analysis of programming languages with ScanVis and Semiotics of Graphic aims at understanding how 2D abstract representations may foster or hinder those transformations. In the figure, the H emerges from the dots because of hard-wired transformation. The study on the relationships between visual variables and the instrumentation of the graphic designer activity aims to help design for the “human perception transform” machinery. The study on ecological representations of time is an example of representations that foster fast and meaningful transformations from data to insights. In the figure, the representation of the writings on the blotting paper enables the reader to directly perceive which letter the writer has spent time on. Again, this perception is hard-wired: designers must live with it and rely on it.
! start hyst drag Press Release Drag > d Drag Release
CStateMachine sm = new CStateMachine(canvas) { CElement toMove = null;
Point2D lastPoint = null; public State start = State() {
Transition press = PressOnShape(">> hyst")}; public State hyst = State() {
Transition drag = Drag(">> drag"); Transition release = Release(">> start")}; public State drag = State() {
Transition stop = Release(">> start")}};
M "hyst"
M "hyst"
Figure 5.7: FromDaDy’s 3D rotation, the ecological representation of time for a healthy and a Parkinson patient, and programming language representations are transformation of data into meaningful representation. They follow principles from hard-wired human perception e.g. selectiveness of luminosity.
Interaction is also a way to enable users to control a transformation of one set of graphics to another to foster perception. As shown previously, the rotating dice is an automatic or controlled smooth transformation of one set of the graphics to another to understand the change. It is also a transformation that helps users perceive new relationships and structures underlying the data through the 3D animation.
5.4.4
Designing and coding transformation
Programming is often concerned with transformations. Interaction programming consists in coding how the data should be displayed and how to translate users’ actions back into operations on the data.
My contributions are principles for the architecture and the code of graphical interactions, using a display transformation, and the inverse transformation to transform users’ actions into operations on the conceptual state. The chain of transformations (conceptual models - high level graphics description - low-level-graphics) can be compiled by a graphical compiler, which is itself a tool to transform a description or specification into executable code (figure 5.8).
The MDPC architecture enabled me to identify, separate and better control the management of the graphicaland the interactive state. The combination of picking views and inverse transformations radically simplifies the code and cancels the need for complex adaptation of analytical code when one adds a new transformation. Interestingly, this architecture makes it possible to leave out the use of a scene-graph, which is a cached state of the graphical scene [Con11]. As such this removal prevents the problems of inconsistency, the complexity of the management of a supplemental data-structure. Though avoiding state seems contradictory with my previous statements, I think that it is not: one should design and implement programs with various combinations of the right concepts, and should not try to hide away from states or “pure functions” for the sake of it.
5.5. TRANSFORMATIONS CONSIDERED AS A LINK BETWEEN INTERACTING AND