The work on affect and its bodily manifestation in psychology and the dance community provides valuable insights that can advance the research in affective computing. In partic- ular, movement notation systems provide a rich tool to capture both kinematic character- istics and expressive qualities of movements in a more compact representation compared to high-dimensional joint trajectories. As FACS1 pushed the research on computational analysis of facial expressions forward, movement notation systems could also advance the computational analysis of affective movements by providing an objective and systematic movement representation.
Burgoon et al. divide movement notation systems into functional and structural ones [6]. Functional systems describe the communicative function of a displayed movement using verbal labels. The Ekman and Friesen formulation of kinesic2 behaviours into five categories (emblems, illustrators, affective displays, regulators, and manipulators) is an example of a functional notation system [51].
• Emblems: culturally shaped body movements (e.g., thumb up).
• Illustrators: actions accompanied with speech to augment the verbalized message (e.g., pointing at an example).
• Affect displays: distinctive bodily expression of different emotion categories.
• Regulators: body movements maintaining the flow of verbal conversation (e.g., head nods).
1Face action coding system (FACS) provides a comprehensive set of action units that can be used to
objectively describe any type of facial movement [49].
• Adaptors: body movements learned and practiced to satisfy personal needs (self- adaptors; occur frequently in private, e.g., head scratching), body language practiced during interpersonal contact such as giving, taking, attacking or being attacked (alter- adaptors; e.g., crossed arm on the chest as a protective posture), movements learned to perform instrumental activities (e.g., driving a car).
Affective movements are classified under the affect display category of the kinesics be- haviours [51], however, other categories of body movements might be involved in the ex- pression of emotion (e.g., self-adaptors can be perceived as a sign of discomfort or anxiety) as these categories are intentionally allowed to overlap.
Structural systems are primarily concerned with the question of what a movement looks like and provide sufficient structural and expressive details for movement replication [6], and are therefore more appropriate for computational affective movement analysis than functional systems. In the following, a review of structural systems that have been applied for computational analysis of affective movements is presented.
Inspired by linguistic notation systems, Birdwhistell proposed a structural movement notation system that parallels phonemic transcription in linguistics [50]. Birdwhistell in- troduced kine (the smallest perceivable body motion, e.g., raising eye brows), kineme (a group of movements with a same social meaning, e.g., one nod, two nods, three nods), and kinemorphs (a combination of kinemes forming a gesture) followed by kinemorphic classes and complex kinemorphic constructs, which are analogous to sentences and para- graphs in linguistics. Birdwhistell used motion qualifiers and action modifiers that define: 1) the degree of muscular tension involved in executing a movement, 2) the duration of the movement, and 3) the range of the movement. The kinegraph is introduced as a tool for notating individual kines and their direction at different body sections. The Birdwhistell system is capable of micro analysis of body movements as its kines capture barely perceiv- able body motion ranging from 1/50 seconds to 3 seconds in duration [50]. Birdwhistell emphasizes the importance of context for inferring the meaning of an observed movement. A Birdwhistell-inspired annotation was used to extract semantic areas in emoticons3 for automatic recognition of their expressed emotions [52].
Delsarte [53] classifies emotion as a form of expression in gestures and divides the body into zones within which mental, moral, and vital components are defined. He identifies nine laws that contribute to the meaning of a movement: altitude, force, motion (expansion, contraction), sequence, direction, form, velocity, reaction, and extension. The Delsarte system has been used for automatic generation of affective full-body [54] as well as hand
and arm [55] movements. In [55], participants’ perception of a set of Delsarte-generated hand and arm movements displayed on an animated agent was shown to be consistent with the Delsarte model prediction.
Recently, Dael et al. proposed BAP (body action and posture), a structural notation system for a systematic description of temporal and spatial characteristics of bodily ex- pression of emotions. Analogous to FACS, BAP introduces 141 behavioural categories for coding action, posture, and function of an observed body movement [26]. BAP segments body movements into localized units in time and describes them at three levels: anatomical (articulation of different body parts), form (direction and orientation of the movements), and functional (behavioural classes categorized in kinesics emblems, illustrators, and ma- nipulators). BAP anatomical and form variables are Boolean (0 for absence and 1 for presence), while functional variables are ordinal (1 for very subtle and 5 for very pro- nounced). BAP was developed using the GEMEP corpus of emotion portrayals. Since the movements are captured from the knees upwards in GEMEP, the current version of BAP does not code whole body postures and leg movements. BAP also does not code dynamic movement characteristics such as velocity, acceleration, and energy. BAP reliability has been demonstrated by assessing intercoder agreement (two coders) on occurrence, temporal precision, and segmentation of posture and action units [26]. There is only a single report on the application of BAP for computational analysis of affective movements at the time of writing this thesis, in which BAP behavioural categories are employed for recognition of 12 affective states encoded in 120 movements demonstrated by 10 actors [56]. Recently, AutoBAP has been proposed for automatic annotation of posture and action units based on BAP anatomical and form (and not functional) coding guidelines [57].
The Laban system is a prominent example of a structural movement notation system, which was developed for writing and analyzing both the structure and expressivity of movements in dance choreography [58,59]. The Laban system has four major components: Body, Effort, Shape, and Space. Body indicates the involved body parts, and the sequence of their involvement in the movement. Space defines where the movement is happening, and the motion directions of the body and body parts. Shape characterizes the bodily form, and its changes in space. Effort describes the inner attitude toward the use of energy. In other words, Body and Space describe What one does through a movement, whereas Effort and Shape describe How the movement is performed [1].
Effort and Shape are the most relevant Laban major components for the study of affective movements. Bartenieff presents Effort and Shape as a complete system for the objective study of movements, from behavioural and expressive perspectives [1]. Effort has four bipolar semantic components: Weight, Time, Space, Flow (see Table 2.1), and Shape has three components: Shape Flow, Directional, and Shaping/Carving (see Table
Table 2.1: Laban Effort components adapted from [1] Extremes Example
Space: Attention to sur- roundings
Direct Pointing to a particular spot Indirect Waving away bugs
Weight: Sense of the im-
pact of one’s movement Light Dabbing paint on a canvas Strong Punching
Time: Sense of urgency
Sustained Stroking a pet Sudden Swatting a fly Flow: Attitude toward bod-
ily tension and control Free Waving wildly
Bound Carefully carrying a cup of hot liquid
2.2). Computational Laban analysis has been carried out for movement recognition (e.g., [60]), and generation (e.g., [61]), and to relate Laban components to low-level movement features e.g., velocity and acceleration [62,63], and different affective expressions [64].
2.2.1
Discussion
For automatic affective movement recognition and generation, there is a need for consistent and quantitative description of movements, leading to a preference for structural notation systems that provide a fixed number of distinct movement descriptors such as the Laban system. However, despite their proven suitability for movement coding, except for BAP, the structural notation systems do not explicitly provide quantitative measures, which is perhaps the main barrier to their application in computational movement analysis. In addition, the extensive attention to microanalysis (e.g., Birdwhistell system [50]), and the need for special training (e.g., Laban system) hamper their adoption in affective computing. Furthermore, some structural notation systems require the coder to infer the meaning or function of an observed movement (e.g., Delsarte [53]). The correspondence between movements and affective expressions is not transcultural and transcontextual, and there may be idiosyncratic, gender-specific, or age-specific differences in affective movements [65]. Such movement/affective expression discrepancies result in a drawback for the notation systems that code the meaning or function of an observed movement. Moreover, the amount and intensity of an affective expression is important for computational analysis; hence, the preference for structural notation systems that code such information (e.g., the
Table 2.2: Laban Shape components [2].
Elements Example
Shape Flow: is self-referential and defines readjustments of the whole body for internal physical comfort.
Growing Self-to-self communication, stretching to yawn
Shrinking Exhaling with a sigh Directional: is goal-oriented and
defines the pathway to connect or bridge to a person, object, or location in space.
Arc-like Swinging the arm forward to shake hands
Spoke-like Pressing a button
Shaping/Carving: is process- oriented and is the three di- mensional “sculpting” of body oriented to creating or experiencing volume in interaction with the environment. Molding, Contouring, or Accommodating Cradling a baby Laban system).
Among the above structural systems, the Laban system is the most popular notation system [8] and enables 1) consistent movement representation in terms of a fixed number of distinct descriptors and 2) studying encoded emotions and their intensity in addition to kinematic characteristics of the movements. However, the Laban descriptors are qualitative and to enable their application in computational analysis, they need to be first quantified. In collaboration with a certified motion analyst (CMA), this thesis develops and vali- dates a quantification for Laban Effort and Shape descriptors, which is then used to design an approach for automatic affective movement generation (Chapter 5).