VoiceDub: Automatic Lip-Synchronization

Film dubbing is a common and expensive process. Dubbing can become necessary in situations where the original recorded sound doesn't have the necessary quality, or when the actor's voice does not convey the intended effect, or still when the original film's language must be translated. In all of these situations studio sound recordings are made after the capture of the images, that need to be synchronized with the movements of the actor's lips. In the case of language translation the process becomes more difficult, since the synchronization with the lip movements is harder to achieve. This work intends to analyze this last situation and to develop tools to help minimizing the synchronization problem.

The VoiceDub Application

Currently, the most common lip-synchronization process used by recording studios is by trial and error: diverse recordings are made until the dub utterance durations are as close as possible to the original ones. In order to automate this procedure we use the concept of distance between visemes together with a cost function to measure the synchronization error. Using this metric, the alignment between the phone sequences of the original and of the dubbing utterance can be computed using an approach based on finite state transducers. Based on this alignment, the durations of the phones of the dubbing utterance can be adjusted to match the original ones and thus producing a better synchronization with the actor's lip movements. Next are presented some time-scaling transformations for a few examples of the test corpus, were it's possible to verify the efficiency of this application.

A Treta Continua

This scene is from the comic Portuguese play A Treta Continua, performed by actors José Pedro Gomes and António Feio, with non-professional dubbing and with very bad lip-synchronization.

avi mpeg4	avi mpeg4	avi mpeg4
Original (Portuguese)	Non-Professional Dubbing (English)	After VoiceDub (English)

Harry Potter and the Prisoner of Azkaban

This scene is from the Warner Bros. Pictures movie Harry Potter and the Prisoner of Azkaban with professional dubbing and with traditional lip-synchronization.

avi mpeg4	avi mpeg4	avi mpeg4
Original (English)	Professional Dubbing (Portuguese)	After VoiceDub (Portuguese)

Brother Bear

This scene is from the Walt Disney movie Brother Bear with professional dubbing and with traditional lip-synchronization.

avi mpeg4	avi mpeg4	avi mpeg4
Original (English)	Professional Dubbing (Portuguese)	After VoiceDub (Portuguese)

Contacts

• Carlos Mendes