4-2). The inventory has been tested on new video-recorded material of episodes of free play whose results are discussed in Section 4-7.
Even though the psychological state of children (i.e. mood, emotions) during interaction is a very relevant variable, we do not include this dimension within the inventory. Coding emotional states would incorporate a higher level of interpretation than the rest of observable interactive behaviors in our system. In addition, children facial expressions are seldom difficult to distinguish and sometimes are not observable from the observer’s position, due to the dynamics of children when playing freely with Pleo and the constrains of the setting (see Section 4.5.4.).
The rest of this section is structured as follows: in the first subsection the more interesting antecedents of behavior catalogues used to describe and measure children’s interactive behavior with social robots are summarized in tables (Tables 4-7 to 4-10) and briefly discussed.
Secondly, the categorization criteria are elaborated and discussed. Thirdly the ethogram is first presented in a summary table (Table 4-12) and afterwards the behaviors are described with the rules of coding, when required, to facilitate the register and coding (Table 4-13). The descriptions are followed by some photographic examples of the specific actions from the videos obtained in our studies on children interacting with Pleo (Figures 4-14 to 4-23).
4.5.1. Antecedents
In 2002 Dautenhahn reported a comparative observational study with children with ASD interacting with a mobile robot -without specific resources for interacting socially- and with a toy truck. The research had a methodological focus and discussed the multimethod approach for analyzing the interaction and communication of children with autism. This position is founded on “the belief that different quantitative and qualitative analysis techniques are necessary to fully assess and appreciate the communication and interaction competencies of children with autism” and thus to inform robots’ design (K Dautenhahn et al., 2002). The trials were carried out in an experimental setting that also involved adults. The observational data were analyzed
using a set of fourteen criteria, broken into two general categories, Action/Behavior and Verbal.
The first category consists of eye gaze, eye contact, operate, handling, touch, approach, move away and attention. The second category consists of vocalization, speech, verbal stereotype and repetition and the analyses proposed was conversational analyses (Table 4-7). Unfortunately, only preliminary results on eye gaze from the Action/Behavior category were reported
Table 4-7 Categories system to measure children with autism’s behavior towards a mobile robot (K Dautenhahn et al., 2002)
Categories Behavior Definition and Examples I Action/Behavior Eye Gaze Direction of gaze
Eye Contact Child making eye contact with a person, gazing at the front of the toy truck, or the heat sensor of the robot which has an "eye-like" appearance
Operate Manipulating an object to make it work, e.g. an IR sensor in order to control the robot
Handling Picking up, pushing, etc. Includes an element of inquisitiveness, including pressing buttons
Touch Physical contact, child initiated
Approach Moving towards. Must be a deliberate movement Move Away Must be a deliberate movement away from object
II Verbal Vocalization Sounds such as yells, mumbling, including whistling (start of vocalization) Speech Word utterances, not necessarily coherent, but a string of words (start of speech) Verbal Stereo Echolalia, non-speech sounds with repetition or without clear purpose
Repetition Any behavior or action which can be grouped and sections repeated, specifically
"autistic behaviors" such as spinning wheels or other distinct repetitive behaviors that autistic children often show
Attention The apparent focus of the child's attention, e.g. robot or toy truck
Other Other Actions that are as yet unclassified, or notes (for example reactions/interactions to people, distress/boredom of child, symbolic play such as stories/play, etc.) Blank No or very little visible child action or introspective behavior without external
purpose, e.g. sitting almost motionless
Kahn’s (Kahn, Jr. et al., 2006) comparison study between interactive behavior towards the pet-robot AIBO and towards a stuffed dog focused on reasoning and behavioral interaction. They studied children’s conceptions of biological entities and their robotic counterparts, from a developmental perspective. The research question was whether children act and think of a robotic pet as if it was alive, and if these behaviors and cognitions changed over time. For the intra-subject design they constructed a categories system of 6 overarching categories to measure and compare how children behave towards the robotic pet AIBO and a stuffed dog with an emphasis on social agency and moral standing attributions (Table 4-8).
Table 4-8 Coding categories of children’s interactions with robotic-dog AIBO and a stuffed dog (Kahn, Jr. et al., 2006)
Behavioral Category Definition and Examples
1. Exploration Anatomy Check Reference to the child's visual or tactile exploration, manipulation, inspection, pointing, and feeding of the artifact. E.g. child explains to the interviewer that AIBO is a boy while inspecting the hindquarters of AIBO. embracing and one‐way verbal greetings to the artifact. E.g. child squeezes the stuffed dog in a big hug.
4. Mistreatment Rough handling Reference to the child's behavior showing disregard for the artifact, including rough handling (e.g. hitting, squishing) and throwing. E.g.
child swings the stuffed dog overhead and then thumps it to the floor.
Thumping
Throwing
5. Endow Animation Vocalize Reference to the child enlivening the artifact in order to perform a behavior or action with it, including making sounds and moving the artifact around. E.g. child throws the bone and says "Fetch!" Then child
According to Kahn’s findings and methodology, Melson carried out another comparison study to investigate the reasoning about and interactions with the robotic pet AIBO and a live dog, an Australian Shepherd (Melson, Kahn, Beck, Friedman, et al., 2009). From a development perspective, Melson’s cross-sectional study covered children from 7 to 15 years old. To study children interactive behavior they proposed a coding scheme with three categories for social behavior -social touch, verbal engagement and attempts at reciprocity-, another category for children exploration of the robot as an artifact and the third one to measure the distance during interaction (Table 4-9).
Table 4-9 Coding scheme of children’s behaviors toward a robot dog and a live dog (Melson, Kahn, Beck, Friedman, et al., 2009)
Behavior Definition and Examples
Exploration as artifact Instances of treating the target dog as an artifact or machine (e.g. poking, shaking) Affection Instances of affection (e.g. hugging, petting kissing stroking)
Attempts at reciprocity Attempts at (to engage it in) reciprocal interaction (e.g. offering a ball, talking to, motioning to), verbal attempts, such as commands (e.g. Come!) or questions (Do you want to play?)
Apprehension Weariness
For systematic observation of children’s behavior with Pleo our research team constructed and applied a coding scheme inspired in Kahn’s and Melson’s works adapted to our research focus on children perception of social agency, investigating both children’s interactive behavior and their judgements and reasoning. Table 4-10 presents the overarching categories and the behavioral units along with a tentative correspondence with Kahn’s categorization (Table 4-8).
Only Physical contact, Gaze, Grooming and Show Something were coded and analyzed (highlighted in the Table 4-10) (Heerink, M., Díaz-Boladeras, M., Albo-Canals, J., Angulo, C., Barco, A., & Casacuberta, 2012).
Table 4-10 Coding scheme for children’s behaviors toward Pleo during free play in pairs (Heerink, M.et al, 2012)
Category Behavior Analytic Category
Emotions Enjoyment
Boredom
Frustration
Neutral
Fear IV Apprehension
Verbal Vocalization
Speak to Pleo II Attempt at reciprocity
Speak to play‐mate
Speak to adult
Speak (other)
Distance Within their grasp
Beyond their grasp
Physical contact Lift up III Exploring as artifact / I Affection
Hug I Affection
Pad I Affection
Stroke I Affection
Let down I Affection
Hold by the tail I Affection (negative)
Hit I Affection (negative)
Handle III Exploring as artifact
Manipulation III Exploring as artifact
Gaze Look at Pleo’s area Orientation to interaction
Look at other Orientation others
Look at adult
Eye‐contact Pleo
Other Interactive behavior Show something II Attempt at reciprocity
Grooming I Affection
Note. Only the shaded units were analyzed in the cited study.
In addition to the antecedent interactive systems to measure interaction between children and social robots, we cite here the coding scheme elaborated and applied by Millot (Millot et al., 1988) to study systematically children’s interaction with dogs. He proposed three overarching categories applicable to group children and dogs’ behaviors: Threaten and aggression, Appeasing and liking and Retreating. It is to notice that the authors excluded explicitly the feeding behaviors. We consider that this attempt to analyze simultaneously both child and dog behavior in the course of the interaction is the most inspiring way to conceptualize and measure the interactive behavior of a child and a pet dyad, analyzing the contingences between antecedent and subsequent behaviors, taking child and dog both as emitters and receivers.
Table 4-11 Coding scheme of child and dog interactive behaviors (Millot et al., 1988)
Categories Behaviors
Children Dog
I Threaten and aggression Threatening the dog Biting, trying to bite
Hitting the dog Barking, growling
Vigorously throwing an object at the dog Pushing the dog away with arms or legs
Pulling the dog’s tail, hair or paw
II Appeasing and liking Patting the dog Approaching the child
Putting the hand on the dog Putting its muzzle
Leaning, squatting or lying beside the dog Sniffing, taking an object presented or given by the child
Stroking the dog Giving an object grasped or solicited by the child
Kissing‐or hugging the dog
Giving an object to the dog
Non‐verbal soliciting
Calling or speaking to the dog
III Retreating Retreating patterns Retreating patterns
4.5.2. Categorization Criteria
Inspired by Kahn’s model (Kahn, Friedman, Freier, & Severson, 2003, 18) and according to the theoretical assumptions and our previous work, the catalog of children’s interactive behavior is separated into two main sections Handle as an Artifact and Social Interaction, separation based on the degree of socialness involved in the intentional behavior towards the robot (Table 4-11) (Heerink, M., Díaz-Boladeras, M., Albo-Canals, J., Angulo, C., Barco, A., & Casacuberta, 2012).
Handle as an artifact encompasses child's inquisitive behaviors of visual (Exploration)- and tactile-manipulative (Manipulation) investigation, putting objects in and out Pleo’s mouth (Put in the Mouth, Take from the Mouth), moving the robot from one place to another (Displace) as well as rough behaviors towards Pleo (Rough Manipulation). Finally one container unit was added to code other kind of no-rough and no-inquisitive tinkering of the robot without displacement (Other).
Social Interaction includes children’s affiliative (e.g. petting, kissing) and no affiliative behaviors (e.g. hitting) with or towards Pleo conveying a social emphasis and implying –to some extend- the attribution to the robot of the subjective entity of a living creature.
Social Interaction is separated in sub-sections based on the valence (attraction vs. aversion) of the rapport shown (i.e. affiliative or pro-social vs not conflict or agonistic behaviors). The sub-categories are Giving affection and Attempt to reciprocity-
According to the intimacy axis, Giving affection is subdivided in turn into two categories:
Substantial contact (i.e. physical contact with children’s chest, head or stomach), Other Contact (i.e. physical contact with hands and fingers) and Affectionate talk, nice and pleasant verbal behavior addressed to Pleo (e.g. baby-talk).
Attempts at reciprocity are affiliative behaviors that tend to obtain a contingent behavior from Pleo. They are social bids addressed to engage Pleo in reciprocal interchanges mainly in providing resources (e.g. feeding), attending its needs or playing. Attempts at reciprocity imply not just responding to the robot actions but expecting the robot to respond in kind based on the previous motioning behavior, verbal directive or offering (Kahn, Jr. et al., 2006). Attempts at reciprocity is a central concept in our model as long as are expressions of children’s theory of Pleo’s mind, their expectancies on Pleo’s capabilities and, in general, of the illusion of pet-likeness (i.e. show a ball and roll it up in the believe that maybe Pleo is going to engage in a catch and fetch play). This category encompasses 6 subcategories: Attention seeking, Present, Offer, Feed, Cuddle and Social Bids.
The category Agonistic includes all behaviors showing disregard or even aggressive and punitive behaviors towards Pleo (Hit and Mistreat) and withdrawal behaviors (Defense).
4.5.3. Inventory
Table 4-12 Children’s interactive behaviors towards Pleo.
Categories, subcategories and behavior units
Table 4-13 Children’s interactive behaviors towards Pleo description
1. ENGAGEMENT
1.1. Handle as an artifact (Fig. 4-16)
1.1.1. Exploration Without physical contact the child visually examines carefully the Pleo’s body or performance. Is a visual inspection observing and checking that implies an element of inquisitiveness. Differs from look at behavior because a postural adaptation to better observe is performed.
1.1.2. Manipulation Manipulation of the robot as touching, pushing or moving parts of Pleo’s body. Handling in order to allow the child checking, inspecting or observing more thoroughly or to explore with their actions Pleo’s functionalities or responses without social meaning (e.g. turn on/off, open the batteries tap). Not necessarily implies physical contact (e.g. passing the hand before the nose’s sensor).
1.1.3. Handling Physical manipulation, without the sense of inquisitiveness with contact with fingers or hands at any part of Pleo body, as touching (i.e. contact with fingers or hands any part of Pleo body), poking (i.e. prod or push with the finger or something narrow or pointed like a learning stone); pinch (i.e. squeeze or compress between the finger and thumb).
1.1.3.1. Mouth Actions related to Pleo’s capabilities of opening the mouth and taking it and keeping between the jaws.
. 1.1.3.1.1. Put in the mouth Introduce objects or a finger between Pleo’s jaws 1.1.3.1.2. Take from the mouth Remove and object or finger from Pleo’s mouth
while Pleo is mouthing it, like taking it from it.
1.1.3.2. Displace Actions on Pleo that results in ostensible displacement (e.g. pull, push) 1.1.3.3. Other Physical manipulation not involving mouthing nor displacement
1.1.4. Rough Manipulation Brusque physical manipulation or handling, misuse with disregard of possible damage (e.g. separating forcefully Pleo’s jaws).
1.2. Social Interaction
1.2.1. Giving Affection Child’s actions towards Pleo addressed to content it, to influence its mood or wellbeing, to give it pleasure or to make it feel good with or without physical contact, regarding or treating Pleo as an object of affection.
1.2.1.1. Substantial Contact Actions that implies close contact with child’s chest, stomach, lap, legs, arms or shoulders. The most intimate behavior is maintaining Pleo stomach with stomach.
1.2.1.1.1
.
Press to bosom Pleo is hold against the child’s body, totally suspended and pressed tightly to his/her chest, stomach, shoulder or neck.
(Fig. 4-17)
1.2.1.1.2
. In Lap Pleo is lifted and placed in the child’s lap or legs. (Fig. 4-18)
1.2.1.1.3
. Hug One or both arms around Pleo’s body the child clasps it tightly with the arms, embraces, and wraps by the arms, without lifting it from the ground or surface where is placed. While hugged Pleo may be hold on the child’s lap or elsewhere as in other’s lap, on the floor or on a table. If there is full contact between child’s and Pleo’s body then the behavior is considered Press to bosom
1.2.1.1.4
. Carry Pleo is picked up, supported and transported to another place by holding in the arms (different from pushing away). (Fig. 4-18)
1.2.1.2. Contact Other Affectionate physical contact with fingers, hands or face.
(Fig. 4-19)
1.2.1.2.1 Pet Touch lovingly, affectionately or tenderly as caressing (i.e. with the pads of fingers, palm of hand, back of the hand, moves along back and forth in continuous contact on back, tail, top of head), scratching (i.e. using fingertips or fingernails to gently rub the chin, particularly where the jawbone connects to the skull), padding/tapping (i.e. strike lightly or gently (repeatedly) with something flat, as with a paddle or the palm of the hand on back, tail, top of head). (Fig. 4-20)
1.2.1.2.2
. Kiss Contact Pleo with the lips or any other part of the face/head as cheek, forehead or chin. (Fig. 4-21)
1.2.1.2.3
. Groom The child puts Pleo on dresses or ornaments, makes Pleo neat or tidy. Cleaning, brushing, removing pieces of dust, and any other manipulation that tend to improve Pleo’s looks and tidiness.
(Fig. 4-22)
1.2.1.2.4
. Touch Other gentle touch or manipulation with physical contact with any part of Pleo’s body, such as hold (i.e. exert pressure with the fingers of one hand or with two hands) to make Pleo look at you.
Fig. 4-23)
1.2.1.3. Affectionate talk Child addresses Pleo in a monologue in a nice or gentle way, seldom using baby-talk, asking questions, appreciative remarks, soothing speaking.
1.2.2. Attempts at reciprocity Socially interactive behaviors of reciprocal nature, in which the child expect a response, a reciprocal interaction. The child not only responds to the artifact, but expects the artifact to respond based on their behavior, verbal directive, or offering (e.g. Pleo walks ahead, child observes Pleo’s behavior and puts a piece of food in front of Pleo and says Come, get it). (Fig. 4-24)
1.2.2.1. Attention seeking With or without physical contact the child attempts to getting the attention or awaking Pleo. When there is physical contact (i.e. hitting or shaking Pleo) it should be gentle and amiable if not the behavior is considered Mistreat. Other attention seeking behaviors are snap fingers, wave, whistle, clap hands, bang on the table or on the floor.
1.2.2.2. Present Bring an object, part of the body close to Pleo’s eyes, mouth, nose or chin as showing it to Pleo as expecting it detects it and responds consequently.
1.2.2.3. Offer Present a piece of food or any object (or a part of the body) close to Pleo’s face (i.e. mouth, cheek, eyes) while looking at Pleo and maintain this position for more than 2 seconds or until Pleo takes it.
1.2.2.4 Feed Put a piece of food or any object between Pleo’s jaws after the behavior of Offer
1.2.2.5. Cuddle Hold Pleo with full contact of Pleo’s stomach to partner’s stomach/
chest/shoulder as expecting Pleo’s to nestle and/or calm down and/or fall asleep.
1.2.2.6 Social Bids Use the arms, hands, and/or fingers or verbal utterances or a combination to communicate and order, direction, request as expecting a contingent response from Pleo based on conventional social exchanges – interpersonal and with pets- as waving hello, greeting, showing numbers with fingers, waving goodbye, pointing, beckoning.
1.2.3. Agonistic Child's behaviors treating Pleo roughly, badly or abusively with disregard to possible damage or even with the intention to harm/damage it (e.g. hitting, squishing throwing). Implies violence, punitive intention, aggressiveness.
1.2.3.1. Aggression (Fig. 4-25) 1.2.3.1.1
.
Hit Deal a blow or stroke to any part of Pleo’s body deliberately with fingers or hands or with an object (e.g. learning stone).
1.2.3.1.2 Mistreat Any other action that implies violence as force feeding (i.e. separate Pleo’s jaws by force or even violence while putting into an object or pushing an object into the mouth), held by the tail (i.e. Pleo totally suspended by the tail and eventually shaken), force or immobilize (i.e. restrain or restrict Pleo’s movements grasping by force or holding it tight or with violence), throw (i.e. the child forcefully flings Pleo through the air, but not just a simple drop).
1.2.3.2. Defense Actions of avoidance and withdrawal ranging from simply startling or leaning away from Pleo to getting up and leaving.
1.3. Attentiveness Without taking part in the interaction, the child looks at Pleo, Pleo’s stuff or at someone who interacts with Pleo, and/or talks about Pleo.
2. DISENGAGEMENT
2.1. Refuse to Interact Child is either passively non-responsive or actively declines interaction with Pleo.
2.2. No interaction Inactivity, inattentiveness, attention focused in other things or event, engaged in other activity or objects.
Figure 4-16 Handle as an artifact
a) Exploration, b) to d) Manipulation
a) b)
c) d)
a) b)
c) d)
Figure 4-17 Giving Affection > Substantial contact a) to d) Press to the bosom
a) b)
c)
Figure 4-18 Giving Affection > Substantial contact a) and b) In lap, c) Carry
Figure 4-19 Giving Affection > Contact Other> Pet
a) b)
c) d)
a) b)
c) d)
Figure 4-20 Giving Affection > Contact Other > Pet> Scratch
Figure 4-21 Giving Affection > Contact Other> Kiss
a) a) bis
b) b) bis
c)
a)
Figure 4-22 Giving Affection > Contact Other > Groom
a) b)
c) d)
Figure 4-23 Giving Affection > Contact Other >Touch a) Poke Head, b) Touch tail, c) Pinch back skin, d) Scratch
a) b)
c) d)
Figure 4-24 Attempts at reciprocity a) Present b) Offer, c) and d) Social Bids
a) b)
Figure 4-25 Agonistic > Aggression a) Hold by the tail; b) Hit on the head