España

3.2.1 Proprioception

The robot is equipped with a variety of sensors that enable it to perceive the properties of objects through a large number of sensory modalities. Each Barrett WAM has built-in sensors in the joints that measure joint angles and motor torques at 500 Hz. In addition to joint-torques, the strains and positions of the fingers of the Barrett Hand can also be measured.

The newer hand design (BH8-280), which is placed on the robot’s right arm, also provides the exact torques applied at each finger joint in real time. Finally, the robot’s right hand is also equipped with a Force-Torque sensor at the wrist that measures 6-DOF forces and torques at the robot’s end effector. Collectively, these sensory signals form the robot’s proprioceptive or haptic sensory system.

DepthRGB

Figure 3.4 Example RGB images and their corresponding depth images taken by the 3DV Systems’ ZCam that is mounted on the robot’s head.

3.2.2 Vision

The robot’s primary visual sensors consist of two Logitech webcams that capture 640 × 480 RGB images. Each of the webcams is mounted on a 2-DOF pan-tilt base unit, embedded in the robot’s head, allowing the robot to control the gaze direction of each eye. The two pan axes are independent of each other, while the tilt axes are coupled.

The robot also has a ZCam, developed by 3DV Systems, which captures 640 × 480 RGB images as well as 320 × 280 depth images. As seen in Figure3.1, the ZCam is mounted on top of the robot’s head. Figure3.4 shows example RGB and depth images capture by the robot’s ZCam as the robot looks at different objects placed on the table in front of it. These images are part of the dataset described in Chapter4.

In addition to the RGB webcam and the ZCam, the robot was recently equiped with a Microsoft Kinect sensor which captures RGB images as well as 3D point clouds. The sensor was mounted on the robots base and pointed towards the table used by the robot to interact with objects. Figiure3.5shows a sample image and its corresponding 3D point cloud captured by the sensor. The images are part of the dataset described in Chapter9.

Figure 3.5 Example RGB image and its corresponding 3D point cloud captured by the robot’s Microsoft Kinect sensor.

3.2.3 Audio

The robot is also equipped with microphones in order to detect auditory feedback that is produced by different objects as the robot interacts with them. The early prototype of the robot used a single Rode NT1-A microphone (see Figures 3.1.a and 3.1.b). That microphone had a cardioid polar pattern with an average self noise of 5 dB. The microphone’s output was routed through an ART Tube MP Studio pre-amplifier. The pre-amplifier supplied 48 volt phantom power to the microphone and sufficient gain was used on the pre-amplifier to provide a suitable input level.

The later version of the robot (see Figure 3.1.c) was equipped with two Audio-Technica U853AW cardioid hanging microphones that were placed inside the robot’s head. The output of each microphone was first routed through an ART Tube MP Studio Microphone pre-amplifier and was subsequently processed through a Lexicon Alpha bus-powered audio interface, which connects to the PC using USB. Sound input was recorded at 44.1 KHz using the Java Sound API over a 16-bit channel. Figure3.6shows an image of the type of microphone that was used as well as a picture of the audio system that was used to route the microphones’ output to the PC.

a) U853AW Microphone b) Audio system

Figure 3.6 a) The Audio-Technica U853AW microphone; b) the two pre-amplifiers (ART Tube MP Studio Microphone pre-amplifiers) and the buspowered interface (a Lexicon Alpha bus-powered interface) that are used to route the microphones’ output to the PC.

Figure 3.7 The artificial fingernail with the three-axis accelerometer sensor, shown by itself (left) and mounted on one of the robot’s fingers (right). The thickness of the fingernail was 0.3175 cm (1/8th of an inch).

3.2.4 Tactile

To perceive object properties through touch, the robot has a vibrotactile sensor in one of its fingers, shown in Figure 3.7. The sensor consists of an artificial fingernail made of ABS plastic and the ADXL345 3-axis digital accelerometer mounted on the EVAL-ADXL345Z evaluation board. Both the accelerometer and the evaluation board were manufactured by Analog Devices.

The accelerometer’s output rate was 400.0 Hz using ten-bit resolution with a range of ±2 g for each axis. The ADXL-345 accelerometer uses an on-board digital low-pass filter, but does not have any analog anti-aliasing filters.

The ABS plastic fingernail was designed with computer-aided design software and printed using a rapid prototyping 3-D printer. The EVAL-ADXL345Z accelerometer evaluation board was mounted on the fingernail, which, in turn, was attached to the middle finger (i.e., F3)

Table 3.1 Summary of the Robot’s Sensory Modalities

Sensory Modality Sensors

Vision 2 RGB Logitech Webcams

1 RGB-D ZCam 1 Microsoft Kinect Proprioception Joint-torque sensors (arm)

Finger strains and torques (hand) Tactile 3-axis fingertip accelerometer

Audio 2 Audio-Technica U853AW microphones

of the robot’s left hand such that its tip protruded from the robot’s finger. When the robot performed a scratching behavior, the vibrations of the fingernail were captured by the attached accelerometer. The accelerometer data were transferred to the PC over a universal serial bus (USB) at 400 Hz using the Arduino Duemilanove microcontroller. The sampling-frequency limitation was due to the limited serial port bandwidth of the Arduino board that was used to communicate with the accelerometer.

3.3 Summary

To summarize, the experimental platform is an upper-torso humanoid robot that has two Barrett WAMs as arms. Each WAM is backdrivable, which allows a human user to quickly program a new exploratory behavior by recording a new joint-space trajectory that can later be replayed by the robot. A table is placed in front of the robot so that it can reach and interact with objects placed on the table.

The robot has sensors that capture data from four different sensory modalities: vision, proprioception, audio, and touch. Table3.1lists the sensors for each modality. The aim of the research described here is to use all of the robot’s sensors to acquire a rich multi-modal object representation. The following chapters describe the experiments that were conducted using the experimental setup described here. They also explicitly mention which subset of the sensory modalities listed in Table 3.1were used in each experiment.