L2F Spoken Language Systems Lab INESC-ID/IST

VoiceDub

Automatic Lip-Synchronization

Film dubbing is  a common and expensive process.  Dubbing can become necessary in situations  where the  original recorded sound  doesn't have  the necessary quality, or  when the actor's  voice does not  convey the intended  effect, or still when the  original film's language must be translated.   In all of these situations studio sound  recordings are made after the  capture of the images, that need  to be synchronized with the  movements of the actor's  lips. In the case of  language translation  the process becomes  more difficult,  since the synchronization  with the  lip  movements  is harder  to  achieve.  This  work intends to analyze this last situation and to develop tools to help minimizing the synchronization problem.


The VoiceDub Application

Currently,  the most common lip-synchronization process used by recording studios is by trial and error: diverse recordings are made until the dub utterance durations are  as  close as  possible to the original ones. In order to automate this procedure we use the concept of distance between visemes together with a cost function to measure the synchronization error. Using this metric, the alignment between the phone sequences of the original and of the dubbing utterance can be computed using an approach based on finite state transducers. Based on this alignment, the durations of the phones of the dubbing utterance can be adjusted to match the original ones and thus producing a better synchronization with the actor's lip movements. Next are presented some time-scaling transformations for a few examples of the test corpus, were it's possible to verify the efficiency of this application.
  1. A Treta Continua
    This scene is from the comic Portuguese play "A Treta Continua", performed by actors "José Pedro Gomes" and "António Feio", with non-professional dubbing and with very bad lip-synchronization.




  2. xvid   mpeg4
    Original
    (Portuguese)

    xvid   mpeg4
    Non-Professional Dubbing
    (English)

    xvid   mpeg4
    After VoiceDub
    (English)

  3. Harry Potter and the Azkaban Prisoner
    This scene is from the "Warner Bros. Pictures" movie, "Harry Potter and the Azkaban Prisoner", with professional dubbing and with traditional lip-synchronization.




  4. xvid   mpeg4
    Original
    (English)
    xvid   mpeg4
    Professional Dubbing
    (Portuguese)
    xvid   mpeg4
    After VoiceDub
    (Portuguese)


  5. Brother Bear - Kenay and Koda
  6. This scene is from the "Walt Disney" movie, "Brother Bear - Kenay and Koda", with professional dubbing and with traditional lip-synchronization.




    xvid   mpeg4
    Original
    (English)
    xvid   mpeg4
    Professional Dubbing
    (Portuguese)
    xvid   mpeg4
    After VoiceDub
    (Portuguese)



This demo page is maintained by Carlos Mendes
Send your comments to: cmdm@l2f.inesc-id.pt