VoiceDub: Automatic Lip-Synchronization: Difference between revisions

Latest revision as of 10:03, 9 July 2007

Film dubbing is a common and expensive process. Dubbing can become necessary in situations where the original recorded sound doesn't have the necessary quality, or when the actor's voice does not convey the intended effect, or still when the original film's language must be translated. In all of these situations studio sound recordings are made after the capture of the images, that need to be synchronized with the movements of the actor's lips. In the case of language translation the process becomes more difficult, since the synchronization with the lip movements is harder to achieve. This work intends to analyze this last situation and to develop tools to help minimizing the synchronization problem.

The VoiceDub Application

Currently, the most common lip-synchronization process used by recording studios is by trial and error: diverse recordings are made until the dub utterance durations are as close as possible to the original ones. In order to automate this procedure we use the concept of distance between visemes together with a cost function to measure the synchronization error. Using this metric, the alignment between the phone sequences of the original and of the dubbing utterance can be computed using an approach based on finite state transducers. Based on this alignment, the durations of the phones of the dubbing utterance can be adjusted to match the original ones and thus producing a better synchronization with the actor's lip movements. Next are presented some time-scaling transformations for a few examples of the test corpus, were it's possible to verify the efficiency of this application.

A Treta Continua

This scene is from the comic Portuguese play A Treta Continua, performed by actors José Pedro Gomes and António Feio, with non-professional dubbing and with very bad lip-synchronization.


avi mpeg4	avi mpeg4	avi mpeg4
Original (Portuguese)	Non-Professional Dubbing (English)	After VoiceDub (English)

Harry Potter and the Prisoner of Azkaban

This scene is from the Warner Bros. Pictures movie Harry Potter and the Prisoner of Azkaban with professional dubbing and with traditional lip-synchronization.


avi mpeg4	avi mpeg4	avi mpeg4
Original (English)	Professional Dubbing (Portuguese)	After VoiceDub (Portuguese)

Brother Bear

This scene is from the Walt Disney movie Brother Bear with professional dubbing and with traditional lip-synchronization.


avi mpeg4	avi mpeg4	avi mpeg4
Original (English)	Professional Dubbing (Portuguese)	After VoiceDub (Portuguese)

Contacts

Carlos Mendes

@@ Line 1: / Line 1: @@
+{{TOCright}}
 Film dubbing is  a common and expensive process.  Dubbing can become necessary in situations  where the  original recorded sound  doesn't have  the necessary quality, or  when the actor's  voice does not  convey the intended  effect, or still when the  original film's language must be translated.   In all of these situations studio sound  recordings are made after the  capture of the images, that need  to be synchronized with the  movements of the actor's  lips. In the case of  language translation  the process becomes  more difficult,  since the synchronization  with the  lip  movements  is harder  to  achieve.  This  work intends to analyze this last situation and to develop tools to help minimizing the synchronization problem.
@@ Line 9: / Line 10: @@
 This scene is from the comic Portuguese play ''A Treta Continua'', performed by actors José Pedro Gomes and António Feio, with non-professional dubbing and with very bad lip-synchronization.
+{| align="center"
+! style="padding-left: 20px; padding-right: 20px;" | [[Image:A_Treta_Continua_original.jpg]]
+! style="padding-left: 20px; padding-right: 20px;" | [[Image:A_Treta_Continua_dobragem.jpg]]
+! style="padding-left: 20px; padding-right: 20px;" | [[Image:A_Treta_Continua_nova_dobragem.jpg]]
+|-
+! style="font-weight: normal;" | [http://www.l2f.inesc-id.pt/demos/VoiceDub/A_Treta_Continua_original.avi avi] [http://www.l2f.inesc-id.pt/demos/VoiceDub/A_Treta_Continua_original.mpeg mpeg4]
+! style="font-weight: normal;" | [http://www.l2f.inesc-id.pt/demos/VoiceDub/A_Treta_Continua_dobragem.avi avi] [http://www.l2f.inesc-id.pt/demos/VoiceDub/A_Treta_Continua_dobragem.mpeg mpeg4]
+! style="font-weight: normal;" | [http://www.l2f.inesc-id.pt/demos/VoiceDub/A_Treta_Continua_nova_dobragem.avi avi] [http://www.l2f.inesc-id.pt/demos/VoiceDub/A_Treta_Continua_nova_dobragem.mpeg mpeg4]
+|-
+! style="font-weight: normal;" | Original<br/>(Portuguese)
+! style="font-weight: normal;" | Non-Professional Dubbing<br/>(English)
+! style="font-weight: normal;" | After VoiceDub<br/>(English)
+|}
+=== Harry Potter and the Prisoner of Azkaban ===
+This scene is from the Warner Bros. Pictures movie [http://www.imdb.com/title/tt0304141/ Harry Potter and the Prisoner of Azkaban] with professional dubbing and with traditional lip-synchronization.
-=== Harry Potter and the Azkaban Prisoner ===
+{| align="center"
+! style="padding-left: 20px; padding-right: 20px;" | [[Image:Harry_Potter_original.jpg]]
+! style="padding-left: 20px; padding-right: 20px;" | [[Image:Harry_Potter_dobragem.jpg]]
+! style="padding-left: 20px; padding-right: 20px;" | [[Image:Harry_Potter_nova_dobragem.jpg]]
+|-
+! style="font-weight: normal;" | [http://www.l2f.inesc-id.pt/demos/VoiceDub/Harry_Potter_original.avi avi] [http://www.l2f.inesc-id.pt/demos/VoiceDub/Harry_Potter_original.mpeg mpeg4]
+! style="font-weight: normal;" | [http://www.l2f.inesc-id.pt/demos/VoiceDub/Harry_Potter_dobragem.avi avi] [http://www.l2f.inesc-id.pt/demos/VoiceDub/Harry_Potter_dobragem.mpeg mpeg4]
+! style="font-weight: normal;" | [http://www.l2f.inesc-id.pt/demos/VoiceDub/Harry_Potter_nova_dobragem.avi avi] [http://www.l2f.inesc-id.pt/demos/VoiceDub/Harry_Potter_nova_dobragem.mpeg mpeg4]
+|-
+! style="font-weight: normal;" | Original<br/>(English)
+! style="font-weight: normal;" | Professional Dubbing<br/>(Portuguese)
+! style="font-weight: normal;" | After VoiceDub<br/>(Portuguese)
+|}
-This scene is from the Warner Bros. Pictures movie, ''Harry Potter and the Azkaban Prisoner'', with professional dubbing and with traditional lip-synchronization.
+=== Brother Bear ===
+This scene is from the Walt Disney movie [http://www.imdb.com/title/tt0328880/ Brother Bear] with professional dubbing and with traditional lip-synchronization.
+{| align="center"
-=== Brother Bear - Kenay and Koda ===
+! style="padding-left: 20px; padding-right: 20px;" | [[Image:Kenay_Koda_original.jpg]]
+! style="padding-left: 20px; padding-right: 20px;" | [[Image:Kenay_Koda_dobragem.jpg]]
-This scene is from the Walt Disney movie, ''Brother Bear - Kenay and Koda'', with professional dubbing and with traditional lip-synchronization.
+! style="padding-left: 20px; padding-right: 20px;" | [[Image:Kenay_Koda_nova_dobragem.jpg]]
+|-
+! style="font-weight: normal;" | [http://www.l2f.inesc-id.pt/demos/VoiceDub/Kenay_Koda_original.avi avi] [http://www.l2f.inesc-id.pt/demos/VoiceDub/Kenay_Koda_original.mpeg mpeg4]
+! style="font-weight: normal;" | [http://www.l2f.inesc-id.pt/demos/VoiceDub/Kenay_Koda_dobragem.avi avi] [http://www.l2f.inesc-id.pt/demos/VoiceDub/Kenay_Koda_dobragem.mpeg mpeg4]
+! style="font-weight: normal;" | [http://www.l2f.inesc-id.pt/demos/VoiceDub/Kenay_Koda_nova_dobragem.avi avi] [http://www.l2f.inesc-id.pt/demos/VoiceDub/Kenay_Koda_nova_dobragem.mpeg mpeg4]
+|-
+! style="font-weight: normal;" | Original<br/>(English)
+! style="font-weight: normal;" | Professional Dubbing<br/>(Portuguese)
+! style="font-weight: normal;" | After VoiceDub<br/>(Portuguese)
+|}
 == Contacts ==
 * [[Carlos Mendes]]
+[[category:Demos]]