AlignDiT

Abstract

In this paper, we address the task of multimodal-to-speech generation, which aims to synthesize high-quality speech from multiple input modalities: text, video, and reference audio. This task has gained increasing attention due to its wide range of applications, such as film production, dubbing, and virtual avatars. Despite recent progress, existing methods still suffer from limitations in speech intelligibility, audio-video synchronization, speech naturalness, and voice similarity to the reference speaker. To address these challenges, we propose AlignDiT, a multimodal Aligned Diffusion Transformer that generates accurate, synchronized, and natural-sounding speech from aligned multimodal inputs. Built upon the in-context learning capability of the DiT architecture, AlignDiT explores three effective strategies to align multimodal representations. Furthermore, we introduce a novel multimodal classifier-free guidance mechanism that allows the model to adaptively balance information from each modality during speech synthesis. Extensive experiments demonstrate that AlignDiT significantly outperforms existing methods across multiple benchmarks in terms of quality, synchronization, and speaker similarity. Moreover, AlignDiT exhibits strong generalization capability across various multimodal tasks, such as video-to-speech synthesis and visual forced alignment, consistently achieving state-of-the-art performance.

Comparison results on LRS3 dataset

Text (Input)	Silent Video (Input)	Reference Speech (Input)	Ground Truth	AlignDiT (Ours)	HPMDubbing	StyleDubber
this is possible
that was powerful
my father worked at this factory he said
they are doing it together
and he said well i just care so deeply about my customers that i would never sell them one of our crappy products
without it we literally can't interface with others
and when i look at the criminal justice system in the united states today i feel the exact same way that i did about the state of
and now i love languages
i'm also on this mission because the way doctors practice medicine continues to
they are gorgeous and headstrong brilliant girls who are a whole lot of brave
you're hearing one presentation after another often representing a group of people a tribe about how
a couple of years ago a group of us in canada decided that we were hitting the limits of what we could accomplish
well imagine you're walking in the forest and you see a bear
i would listen to the heart
my dad has alzheimer's disease
not the wife not the kids
all of you are members of tribes
if you want this experience to live on as something historic then at the reception
well very often we think about political change in defined compartments these days
i know you understand that

Ablation study of modality conditioning method on LRS3 dataset

Text (Input)	Silent Video (Input)	Reference Speech (Input)	Ground Truth	(a) w/o 𝓛_CTC	(a)	(b) w/o 𝓛_CTC	(b)	(c) w/o 𝓛_CTC	(c) AlignDiT
this is possible
that was powerful
my father worked at this factory he said
they are doing it together
and he said well i just care so deeply about my customers that i would never sell them one of our crappy products
without it we literally can't interface with others
and when i look at the criminal justice system in the united states today i feel the exact same way that i did about the state of
and now i love languages
i'm also on this mission because the way doctors practice medicine continues to
they are gorgeous and headstrong brilliant girls who are a whole lot of brave
you're hearing one presentation after another often representing a group of people a tribe about how
a couple of years ago a group of us in canada decided that we were hitting the limits of what we could accomplish
well imagine you're walking in the forest and you see a bear
i would listen to the heart
my dad has alzheimer's disease
not the wife not the kids
all of you are members of tribes
if you want this experience to live on as something historic then at the reception
well very often we think about political change in defined compartments these days
i know you understand that

Ablation study of input modalities on LRS3 dataset

Text (Input)	Silent Video (Input)	Reference Speech (Input)	Ground Truth	Only Video	Only Text	AlignDiT (Ours)
this is possible
that was powerful
my father worked at this factory he said
they are doing it together
and he said well i just care so deeply about my customers that i would never sell them one of our crappy products
without it we literally can't interface with others
and when i look at the criminal justice system in the united states today i feel the exact same way that i did about the state of
and now i love languages
i'm also on this mission because the way doctors practice medicine continues to
they are gorgeous and headstrong brilliant girls who are a whole lot of brave
you're hearing one presentation after another often representing a group of people a tribe about how
a couple of years ago a group of us in canada decided that we were hitting the limits of what we could accomplish
well imagine you're walking in the forest and you see a bear
i would listen to the heart
my dad has alzheimer's disease
not the wife not the kids
all of you are members of tribes
if you want this experience to live on as something historic then at the reception
well very often we think about political change in defined compartments these days
i know you understand that