Sobre el nivel de adopción

The core resource framework provides a set of classes derived from Resource that provide some additional functionality, primarily in the form of implemented draw() functions. It is rare that when an application wishes to provide its own Resource that it will inherit directly from Resource – more often the Image or Drawingderived classes will be used.

The classes Image, Poly, Drawing, and Arrow derive directly from Resource. Also provided are Blackhole, derived from Image; Writing and Circle, derived from Drawing; and AudioCircle derived in turn from Circle. These are shown in Figure5.2: other classes in the hierarchy are provided in plugins and plugin libraries. These eight classes are core because they provide functionality tightly linked to the operation of Cruiser, without pulling in additional library dependencies, while providing a framework for code reuse by plugins; thus working within our design goals (§3.5.1).

Less-visible components supporting the operation of Resource are also found here: the position cache for efficiently inheriting object transformations, Layouts for organising child objects, and Resource bounds tracking.

5.2.4.1 Image

Image, at the left of Figure5.2, is the most used subtype of Resource in Cruiser. Essentially, it represents any on-screen object whose representation is backed by arasterised image. Thus, the majority of its implementation is geared towards efficiently loading, rendering and sharing this image data. Image also serves as a convenient container as it is always rectangular in shape and thus has bounds that are easy to determine.

The most obvious example of an Image is a photograph file provided from disk. However, any rasterised content is supported, including Portable Network Graphics (PNG)

images with analpha channel (useful for interface elements), partial screen captures and images loaded over the network. Classes derived from Image are also able to leverage the functionality provided for still images to implement dynamic images – video and remote

framebuffers – by continuously updating the texture data.

Image implements the texture processing assembly line described in Subsection 3.5.5.1 on page 53, in order to load image data from disk into atexture that may be efficiently rendered by OpenGL. A reload is also supported, where an image initially loaded in low-quality (e.g. for a thumbnail) may be upgraded into full quality for detailed viewing. Default decorations are also provided for Images. These are elements drawn over the image data that indicate the object boundary and ownership (by colour) of an Image. When selected, elements indicating hot spots – areas of the image that when touched activate rotating, resizing or flipping – are also displayed (see §4.2.1). Decorations are rendered efficiently through the use of OpenGL display lists. This is a procedure whereby the sequence of drawing operations comprising the decorations may be remembered by the graphics card and given an identifier. Subsequently drawing the decorations merely requires asking the graphics card to repeat the sequence, using a single function call.

Image manages various texture processing features when loading textures that control quality and efficiency. Anisotropic texture filtering is a feature provided by modern graphics card and is available in OpenGL. It provides high-quality display of textures when they are rendered at an angle to the screen (e.g. when flipping), but is also able to efficiently smooth image data during texture minification. Minification occurs when the texture is rendered on screen smaller than a direct mapping of the texture to screen pixels would provide. Because Cruiser uses very large textures for high-quality photo viewing, minification is very common, so the benefit from anisotropic texture filtering is significant – without it, photos made small on the display would appear blurry. Texture compression and helper functions to load Exif thumbnails or textures cached in the metadata database (as discussed in §3.5.5.1) are also managed by Image.

The OpenGL identifiers provided by the graphics implementation when a texture or display list is created are able to be shared amongst Images. A third, helper function can check whether a particular file has already been loaded, but sharing most commonly occurs when an Image is copied. This is frequent: interface elements are often duplicated (e.g. when providing many poker chips of the same denomination, or when duplicating control objects so each user has a copy), and photographs are often copied from their thumbnail representation (thence triggering an upgrade of both copies, which may then be shared). Texture and display list handles are shared with reference counting.

Helpers Loading can also be assisted by an Exif helper. The actual Exif parsing is provided by an external library and activated by a plugin (§5.5.2). So, an Image may be loaded using any of the following techniques (tried in order, in a background thread):

• if initiated by a copy() of an existing Image, perform member-wise copy of data (uses already-loaded OpenGL texture ID and increments reference count); otherwise,

5.2. Cruiser Core CHAPTER 5. DESIGN

• if loaded by filename, pass the image filename and desired quality to the preloaded_ _helper, which will return a reference-counting handle to the OpenGL texture ID if that file was already loaded with the desired, or better, quality; otherwise

– if high quality (i.e. non-thumbnail quality) is requested, load the image and generatemipmaps from disk; otherwise

• if loaded by filename, pass the filename to the texture_helper, which will re- turn an array of data structures representing arguments to successive calls of the glTexImage2D OpenGL function to load compressed textures that were previously cached in the metadata database; otherwise,

• if not cached, pass the filename to the thumbnail_helper, which checks the file on disk for anExif header and, if found, returns the stored miniature thumbnail as an in-memoryJPEG that is subsequently loaded as if it were the entire file from disk (as in the next technique); otherwise

• pass the filename to the SDL_Image library to decode the file from disk, rescale and generatemipmaps (as described in §3.5.5.1 on page 53).

The first two of these techniques create a reference to an existing texture, which is used immediately once the Image is added to the Environment7 _{and the screen redrawn. The}

other three techniques all result in a sequence of successive arguments to pass to the glTexImage2D OpenGL function, in order to load the set ofmipmaps created into texture memory.

This final step must be done in the main thread, so the array of argument sets is held temporarily in system memory and a low-priority event is queued (§5.2.7) to perform the function calls between screen redraws, in the main thread. The event is given low priority because the step that transfers the texture data from system memory to texture memory can be lengthy – possibly hundreds of milliseconds8 – and high priority events (such as user input) should be processed first to keep the interface responsive.

The texture-loading step is also the step where the graphics card may compress the textures in its own memory, and OpenGL has a facility to subsequently retrieve the compressed textures from the graphics card. If the texture_helper technique was unable to find cached textures in the database, Image now retrieves the compressed textures at this stage; replacing the uncompressedmipmap data in system memory. The (now compressed) texture loading arguments are then passed to a managed background thread, which is able to save the compressed thumbnail textures to the database for later use.

5.2.4.2 Layouts and PositionDependant

A Layout is a mechanism for arranging a collection of Resources on screen. As shown in Figure5.3, it is a class derived from PositionDependant9, which allows the Layout to be informed of manipulations of a parent Resource. Thus a Layout is typically used to arrange objects displayed on the parent Resource, acting as a container ; when the parent is moved, rotated or resized, the collection of objects it contains will inherit the transformation and possibly be rearranged by the Layout.

Non-layouts that derive from PositionDependant are used to inherit transformations for single objects. Arrow and Label are Resources (see Figure A.2for the full inheritance

7_{this is a thread-safe operation so occurs between screen redraws} 8_{but rarely more than a second, and much less for thumbnails}

9_{the spelling dependant (rather than dependent) is deliberate for two reasons: dependant is always a}

PositionDependant

Arrow Label Layout PDAdapter

FlowLayout HistoryLayout QuadTreeLayout

BettingBoxLayout SFTLayout CopierLayout FixedPDAdapter Provided by Plugins and Plugin Libraries

Figure 5.3: Layout class hierarchy (with PositionDependant)

lattice) that work together to implement labelling of Images with annotation callouts – the label should look like it is pinned to the Image. PDAdapter follows the adapter design pattern to allow regular Resources to become dependants, where it is not otherwise a requirement. FixedPDAdapter fixes a Resource relative to the centre of another, such that it cannot be moved by itself.

For example, Writing (described on the following page) can be done directly on an Image, where it will become stuck, using a PDAdapter. It can still be moved around the Image – PDAdapter does not remove Resources’ own abilities to be manipulated (FixedPDAdapter does). If it is moved off the image, it becomes detached; the adapter is discarded and the writing becomes an independent Resource. However, it may be later

attached (on the back) of an image (see §5.2.1.2 on page 102), whereby it will then become

part of a Layout.

Only one layout is supplied in the core. This is in line with the design goals (§3.5.1): the intention is not to create an exhaustive set of functionality, but only to provide a framework with the flexibility to add functionality. The added functionality can then be shared through the use of plugin libraries. The layout provided in core, FlowLayout, is shown in Figure5.3. FlowLayout implements a layout similar to words on a page, with Resources being placed on a new row when they would exceed the width of the container. The name is inspired by java.awt.FlowLayout10. The other layouts in Figure 5.3– HistoryLayout, BettingBoxLayout, SFTLayout (space-filling thumbnail), and CopierLayout – are provided by plugins and plugin libraries (§5.4).

The collaboration diagram for FlowLayout is shown in Figure 5.4. This figure is also provided to give some context for the Top-Level Utilities described in §5.2.3. A Layout contains a collection (coll) of reference-counted Resources and inherits the parent from PositionDependant. FlowLayout extends Layout with a padding and margin in order to arrange Resources in coll within the bounds of the container parent. Using Layout in this manner separates the management required to add and remove items from the layout (and, importantly, the automatic removal of dependencies) from the logic that performs the layout. Layout also provides a thread-safe view on the collection of items, so that new

5.2. Cruiser Core CHAPTER 5. DESIGN FlowLayout Layout PositionDependant Resource link bool frontside alwaysfront RefT< Resource > parent

vector< RefT < Resource > > < RefT < Resource > > r Refmutex rlock refcnt_lock SDL_mutex * mut Refcount refs RefT < Resource > const void * lastAdded lastSettled double padding margin coll unsigned refs

(note: Resource data members are incomplete)

Figure 5.4: Collaboration diagram for FlowLayout

objects can be added from background threads, for example, once an object has finished loading from disk.

5.2.4.3 Writing, Drawing and Polygon

Writing can either be typed text or a stroke, made with a finger or stylus. Typed text is rendered using the polyfonts11library, with a significant improvement added by Cruiser: the first time any font glyph is drawn, its shape is cached in an OpenGL display list so that subsequent drawings of the same letter are very quick. Thousands of characters can be rendered at any size or orientation in a matter of milliseconds this way, in order to maintain a screen redraw rate of more than 50 frames per second.

Strokes are rendered similarly to typed text, with polygons. As a stroke is being drawn, it is rendered using OpenGL’striangle stripprimitive, as shown in Figure5.5 on the facing page. There are some subtleties in the algorithm to deal with special cases, but essentially successive trapezia12 (made of two triangles in the strip) are drawn such that their parallel lines are parallel to the line segment joining each pair of points on the stroke path. The angles of the non-parallel sides are chosen so that they bisect the angle between successive line segments (perpendicular at the ends, where there is only one line segment). Width of the triangle fan can be configured. Once the stroke has been completed, repeating these

11

http://www.gameprogrammer.com/polyfonts/polyfonts.htmlverified 2007-02-13.

12_{Plural of trapezium (British English), meaning a quadrilateral with one pair of parallel sides (or trapezoid}

Figure 5.5: Rendering a stroke using OpenGL’s triangle strip

The dashed line in the centre adjoins points on the stroke path; real strokes have many more points than shown here and segments are typically much wider than they are long; all triangles (not just those solid in

this figure) are drawn in solid colour

calculations would be wasteful, so the points are normalised between ±0.5, a suitable scale is chosen so the normalised result is identical, and the sequence of OpenGL calls are cached in a display list, for efficient rendering.

These techniques are a departure from the usual ways of rendering text and handwriting in an interface. It is traditional for interfaces recording stroke data (e.g. from a stylus) to require the input to be made on a canvas. That is, an object or area of screen that behaves similar to a piece of paper; writing on the canvas changes the pixel matrix that represents the canvas. In some cases it is possible for the canvas to be transparent, such as is used when annotating an entire screen, e.g. when using software packaged with interactive whiteboards. However, activating the canvas is modal – you can either annotate the canvas or manipulate objects behind it, but not both, unless you switch modes. Similarly, text is typically entered in a text box.

In Cruiser, the text glyphs and stroke annotations themselves are objects that can be selected, manipulated, deleted and attached to other objects in the environment. Of course, subtypes can easily be derived from Writing with any of these functions disabled for text that might be static. Rendering strokes as OpenGL primitives allows them to be easily selected, using the graphics-accelerated picking techniques available through the Resource ancestor class.

From Figure5.2, it can be seen that Writing is derived from Drawing. A Drawing is notionally an object that is drawn manually, using OpenGL drawing functions. Foremost, it is distinct from an Image, which may have non-texture decorations, but is always rectangular in shape. The Drawing class provides a drawer object that usesRAIItechniques to establish and clean up transformations in OpenGL’s model-view matrix to provide classes derived from Drawing their own coordinate system, independent of the transformations inherited from Resource (e.g. move, rotate, resize), which are applied automatically. An example use of a Drawer is shown in Listing 5.1 on page 111.

Note that Drawings (including text and strokes) are easily selected and moved around the interface. However, by default they do not have decorations that indicate areas that allow them to be rotated, resized or flipped, as Images do. A derived type can indicate to the gesture framework that these manipulations should be performed instead of move, e.g. depending on the area selected. However, the transformation inheritance provided by Cruiser’s Layouts provides an alternative way to rotate and resize drawings: attachment. Attaching a Drawing to an Image (which may be a simple container, without a texture), and then manipulating the Image lets those manipulations be inherited through the Layout. Subsequently detaching the drawing retains the transformations made to it whilst it was

5.2. Cruiser Core CHAPTER 5. DESIGN

attached.

A Poly is a basic polygon shape, drawn with OpenGL’s polygon primitive. Such a shape could easily be implemented by inheriting from Drawing, but Poly is special. Rather than being a screen object that may be manipulated, a Poly is fixed in space. Furthermore, the points of the polygon are given inscreen coordinates, and automatically translated to world coordinates when the Poly is created (using setworld, described later in this subsection). By default, Polys are also drawn behind any other objects in the environment. Poly is an example of a screen object with a specific task, for which it makes sense to inherit directly from Resource. Specifically, Poly was used to implement an early version of fixed personal spaces (§7.1.6.3).

5.2.4.4 Black Hole

The Black Hole has long been a feature of Cruiser. It is a screen resource for deletion, similar to a trash can. In early versions it was tightly coupled with other core functionality, but new flexibility in Cruiser means that it has been refactored so that it can instead by implemented as a plugin. From a plugin writer’s perspective, it is an example of a Resource that automatically selects all items nearby, and influences objects moved around it. From a user’s perspective it is much more, and is described in Subsection 4.3.1.

Black Hole Details To support the Black Hole, there exist algorithms for determining the layering of photographs around the Black Hole, an algorithm for keeping objects ‘in’ the Black Hole as it is moved and an algorithm/formula for the ‘wormhole’ effect that determines the size of images around the black hole based on the image size, the size of the black hole and the Euclidean distance between the centres of each image and the Black Hole.

For each on-screen object (including the black hole object) we maintain the following state:

• (x, y, z) position of the centre of the object, inworld coordinates(z determines the order objects may be drawn)

• (x, y) position of the touch point, inobject coordinates, if an object is being moved • s, the desired scale of an object, changeable by performing a rotate/resize operation

on the object

• userLock, the identifier of the user moving an object or NO_USER

For objects other than the Black Hole, we calculate the following on each screen redraw: • bhd, the most recently determined distance from the Black Hole, calculated as

bhd= d

√ 2

s , where d is the square of the Euclidean distance between the centre of the Black Hole and the object and s is the current scale of the Black Hole

– An object is said to be in the black hole when bhd< 1.0

– The fringe of the black hole is the circular area around the Black Hole where bhd< 1.0

For objects other than the Black Hole, we also maintain the following:

• residual, an indication of whether an object has been moved into and subsequently

released in the fringe of the black hole, and the value of bhd when that occurred

1 v o i d C i r c l e :: d r a w ( b o o l r e n d e r i n g ) { D r a w e r d ( this , r e n d e r i n g ) ; g l B e g i n ( G L _ T R I A N G L E _ F A N ) ; g l V e r t e x 3 d (0.0 , 0.0 , 0 . 0 ) ; d o u b l e t h e t a = 0; 6 f o r ( u n s i g n e d i = 0; i <= s e g m e n t s ; ++ i , t h e t a -= s i g m a ) g l V e r t e x 3 d ( 0 . 5 * sin ( t h e t a ) , 0 . 5 * cos ( t h e t a ) , 0 . 0 ) ; g l E n d () ; }

Listing 5.1: Circle’s draw member function, using a Drawer

Drawer applies transformations (move, rotate, resize, etc.) from Resource, and automatically returns OpenGL’s model view matrix to its previous state when the draw function exits, so any Drawing is able to work in its own coordinate system regardless of where it appears on the interface; segments and sigma are data members initialised in Circle’s constructor

– if an object enters the fringe of a Black Hole other than by a move (or if the black hole itself was moved such that its fringe encompasses the centre of an object) then its residual_bhd is 1.0

When an object is “in” the black hole, we alter the displayed scale (after processing the