BASES TEÓRICAS

With the aim of avoiding the tedious manual lip synchronisation discussed in the previous section, a significant part of lip synchronization efforts has been devoted to the automation of the process. These efforts can be divided into three major functional groups, namely those driven by text, speech and image. Text-driven lip synchronisation attempts to produce visual and audible phonemes from a printed text. Speech-driven analyses a pre-recorded speech file, extracting the required phonemes from it. These phonemes are then used to produce the visual aspect of synthesized speech. The image- driven approach uses video frames in an attempt to identify the uttered phoneme. Another approach that is worth mentioning is the hybrid approach, which combines text-driven and speech-driven approaches in an attempt to further improve the automation of visual speech.

9.2.1 Text-driven approach

In a text-driven approach, visible and audible speech are synthetically generated from a text. There are two main directions in the text-driven approach. One consists of mapping text to visemes by means of vector quantization, initiated by Waters and Levergood (1993 and 1994). The other utilizes a rule-based system, as described by Pelachaud, Badler and Steedman (1996) and Beskow (1995).

Waters and Levergood (1993 and 1994) relied on speech synthesizers (DECtalk) to interpret the text and provide their system with phonemes. Their system follows the synthesized pronunciation with the corresponding expressions. Time is a critical aspect of this approach. In order to be realistic, the visual and the audible systems need to have a common time base reference. Since the audible system operates at a far higher frequency, the visual system calculates its interpolation in terms of the audible system’s clock. Pelachaud, Badler and Steedman (1996) based their approach on the Facial Action Coding System (FACS) (see, for example, Ekman, Friesen and Hager, 2002) , using a text file as their speech source. In addition, the source text is required to be in its phonetic representation beforehand. Apart from the phonemes, the text would contain affect and its intensity, phoneme timing and intonation structure. The algorithm reads the file and computes the lip shapes. Action Units (AU) (Ekman, Friesen and Hager, 2002) are used for the basic lip shapes. These basic shapes are then modified by taking into account the influence of emotions, coarticulation and other contributing factors. As the animation engine, the authors used Jack – animation software developed at the University of Pennsylvania (Badler, Phillips, and Webber, 1993). Beskow (1995) used a parameterised model based on the one originally designed by Parke (Parke and Waters, 1996). The main differences between Parke’s and Beskow’s model are in the introduction of lip movement

parameters and the tongue. Beskow used two existing systems, namely RULESYS, that converts the plain text into phonemes and other speech data, and GLOVE, to synthesize audible speech. The role of synchronisation between the visual and audible speech is performed by the RULESYS system. Ezzat and Poggio (1998) presented a 2D audio/visual text-to-speech system. The system mapped phonemes to pre-recorded images representing visemes. The images in between two phonemes were achieved by morphing. In the interests of simplicity, coarticulation was ignored.

A pure text-driven approach often forms part of a speech-driven approach. In a speech-driven approach, the speech is often analysed, then converted into a sequence of phonemes, using a text- driven approach thereafter. Speech-driven approaches are explained in the following section.

9.2.2 Speech-driven approach

In a speech-driven approach, the computer first analyses the pre-recorded speech, and then generates visual facial representations. Pioneering the first notable speech-driven automated lip-synchronisation attempt, Lewis (1991) first outlined some of the historical methods. The early naïve approach to lip- synchronisation consisted of opening the mouth in proportion to the loudness of the sound. It is obvious that this approach did not have good results, as pronouncing ‘m’ can be loud, while the mouth hardly opens at all. A more advanced approach involved passing the sound data through a series of filters and producing the animation based on spectra outputs. Although acceptable, this approach, used in the early 1980s, was still unable to produce the fully realistic lip motion. The main drawback was in the spectrum containing the vocal tract and pitch mixed together, while speech depends exclusively on vocal tract data. Lewis’s approach was based on a linear prediction model, and involved the sound being broken down to sound source and vocal tract filtering components. The filtering components were the parts used for lip-synchronisation. Using the linear prediction algorithms, they were converted into a phonetic script. References phonemes were extracted from the script first. Reference phonemes are English language vowels (Figure 107) and the consonants m, s and f.

Figure 107: Mouth positions for the vowels in the English language (Lewis, 1991). The top row are the vowels in hat, hot and the f/v sound. The bottom row are the vowels in head, hit and hoot.

While identifying the vowels using a linear prediction identification approach was not difficult, correctly identifying the consonants posed a significantly greater challenge. For example, when the consonant ‘t’ is at end of a word, the mouth may have to remain open in anticipation of the next word. The model used for the animation was based on Parke’s (1996) parameterised approach. The tongue was not included in the animation. An interesting contribution to the speech-driven lip synchronisation school of thought was made by Brand (1999). He used artificial intelligence principles to train his system on face reaction in response to speech. Using this system, face trajectories were mapped directly to the voice patterns, bypassing the process of decomposition of the sound signal to phonemes. This was followed by animation of the phonemes, taking coarticulation into account. Some authors tried to bypass the intermediate step of analysing the audio information to a sequence of phonemes first. Bondy et al. (2001) described a method of concurrent analysis of audio and video. During the training phase, an audio signal is mapped to lip positions deduced from the video. Once the training is completed, the system is able to analyse new audio speech and retrieve the facial expression mappings accordingly. One of the latest additions to speech-driven animation is the capture and reproduction of other audio-conveyable information, such as intonation (Gutierrez-Osuna et al., 2005).

9.2.3 Image-driven approach

Apart from audio, speech can also be deduced from a video source. A computer can employ machine vision techniques to perform lip-reading from video footage. One of the first notable works in this area was by Ezzat, Geiger and Poggio (2002). Their system was capable of analysing video footage, then synthesising visual speech from the analysed data. Once the initial fifteen minute footage is recorded, the system would spend several days analysing the footage and learning (using gradient descent learning) how to synthesise the speech. During the learning period, no user intervention was required.

Figure 108: Some results of Ezzat, Geiger and Poggio’s (2002) visual speech synthesis approach.

The new video was synthesized using 2D image processing techniques. A multidimensional morphable model (MMM) was used for the generation of expressions, while a trajectory synthesis technique based on regularization was used to generate trajectories within the MMM space. The main disadvantage of this approach was in the difficulty of blending the newly created images into the background: catering for substantial movement of the head, changes in lighting conditions and changes in the viewpoint. Some sample frames produced using this approach are shown in Figure 108. Basu, Oliver and Pentland (2003) designed a system capable of capturing 2D lip motion from a video file, and then animating a 3D lip model in accordance with the analysed data. Finding an accurate way to capture lip movement from video is challenging, partly because there is not much

visual information with which to work. Also, the head seldom remains still. It rotates constantly, leaving a different angle 2D projection of the lips for analysis every time. Also, the contour of the lips is often obscured by lighting, facial hair and other disturbances. The primary way of differentiating the lips from the rest of the face is using their colour content, which is often noisy. Due to all the abovementioned obstacles, it would be extremely hard, if not impossible, to capture accurate lip movement information using only machine vision principles.

As an auxiliary method, Basu, Oliver and Pentland (2003) used statistics of expectancy of posterior behaviour, taking into consideration visible 2D posture. To aid machine vision, sixteen ink marks were plotted on the actor’s lips prior to filming, along with one on the nose as a reference. This initial filming was used to train the system (the actual footage did not have any markings). The details of the computer vision principles utilized are omitted, as they are beyond the scope of this thesis.