A-DMA
Accelerating Diffusion-based Text-to-Speech Model Training with Dual Modality Alignment
[Paper] [Code]
Jeongsoo Choi1*, Zhikang Niu2,3*, Ji-Hoon Kim1, Chunhui Wang4, Joon Son Chung1, Xie Chen2,3
1School of Electrical Engineering, KAIST, South Korea
2MoE Key Lab of Artificial Intelligence, X-LANCE Lab, School of Computer Science,
Shanghai Jiao Tong University, China 3Shanghai Innovation Institute, China 4Geely, China
Abstract. The goal of this paper is to optimize the training process of diffusion-based text-to-speech models. While recent studies have achieved remarkable advancements, their training demands substantial time and computational costs, largely due to the implicit guidance of diffusion models in learning complex intermediate representations. To address this, we propose A-DMA, an effective strategy for Accelerating training with Dual Modality Alignment. Our method introduces a novel alignment pipeline leveraging both text and speech modalities: text-guided alignment, which incorporates contextual representations, and speech-guided alignment, which refines semantic representations. By aligning hidden states with discriminative features, our training scheme reduces the reliance on diffusion models for learning complex representations. Extensive experiments demonstrate that A-DMA doubles the convergence speed while achieving superior performance over baselines.
Contents
Model Overview
Figure 1. Overall training framework. The model uses masked speech, noisy speech, and transcript to predict the masked part of the speech via flow matching. The SSL model and CTC layer are utilized only during training, with the SSL parameters frozen throughout the process.
Performance over Training Steps
Figure 2. The WER and SIM results showing the effectiveness of the proposed dual modality alignment.
* Each text below each audio is ASR transcribed and normalized.
Text: And to think we can save all that misery and despair by the payment of a hundred and fifty dollars!
| Prompt Speaker | Ground Truth | Training Steps | F5-TTS + A-DMA (Ours) |
F5-TTS |
|---|---|---|---|---|
|
and to think we can save all that misery and despair by the payment of a hundred and fifty dollars |
100k |
and to think we can save all that misery and despair by the payment of a hundred and fifty dollars |
and to think when (missing) save all those re a despair and disbam a payment of (missing) hundred and fifty dollars |
|
|
200k |
and to think we can save all that misery and despair by the payment of a hundred and fifty dollars |
and to think we can save all that misery and despair by (missing) payment of a hundred and fifty dollars |
||
|
300k |
and to think we can save all that misery and despair by the payment of a hundred and fifty dollars |
and to think we can save all that misery and despair by the payment of a hundred and fifty dollars |
||
|
400k |
and to think we can save all that misery and despair by the payment of a hundred and fifty dollars |
and to think we can save all that miserisery and despair by the payment of a hundred and fifty dollars |
Text: Why, it's Goliath as usual! they both cried, peering in.
| Prompt Speaker | Ground Truth | Training Steps | F5-TTS + A-DMA (Ours) |
F5-TTS |
|---|---|---|---|---|
|
why its goliath as usual they both cried peering in |
100k |
why its goliath as usual they both cried peering in |
why its goleth as usual thoth (missing) cried peering in |
|
|
200k |
why its goliath as usual they both cried peering in |
why its goliath as usual they both cried peering in |
||
|
300k |
why its goliath as usual they both cried peering in |
wide scull i ath as usual they both cried peering in |
||
|
400k |
why its goliath as usual they both cried peering in |
why its goliath as usual they both cried peering in |
Text: The first of our vague but indubitable data is that there is knowledge of the past.
| Prompt Speaker | Ground Truth | Training Steps | F5-TTS + A-DMA (Ours) |
F5-TTS |
|---|---|---|---|---|
|
the first of our vague but indubitable data is that there is knowledge of the past |
100k |
the first of our vague but indubitable data is that there is knowledge of the past |
the first of our very vague doodledata (missing) that there is knowledge of the past |
|
|
200k |
the first of our vague but indubitable data is that there is knowledge of the past |
the first of our vague but indubitable data is that there is knowledge of the past |
||
|
300k |
the first of our vague but indubitable data is that there is knowledge of the past |
the first of our vague but indubitable data is that there is knowledge of the past |
||
|
400k |
the first of our vague but indubitable data is that there is knowledge of the past |
the first of our vague but indubitable data is that there is knowledge of the past |
Text: It has no beauty whatsoever, no specialty of picturesqueness; and all its lines are cramped and poor.
| Prompt Speaker | Ground Truth | Training Steps | F5-TTS + A-DMA (Ours) |
F5-TTS |
|---|---|---|---|---|
|
it has no beauty whatsoever no specialty of picturesqueness and all its lines are cramped and poor |
100k |
it has no beauty whatsoever no specially of picturesqueness and all its lines are cramped and poor |
it has no divided of her no special trisquinus (missing) and all its line crumped (missing) and poor |
|
|
200k |
it has no beauty whatsoever no specialty of picturesqueness and all its lines are cramped and poor |
it has no beauty whatsoever no specie of picturesqueness and all its lines are cramped and poor |
||
|
300k |
it has no beauty whatsoever no specialty of picturesqueness and all its lines are cramped and poor |
it has no beauty whatsoever no specie of picturesqueness and all its lines are cramped and poor |
||
|
400k |
it has no beauty whatsoever no specialty of picturesqueness and all its lines are cramped and poor |
it has no beauty whatsoever no specie of picturesqueness and all its lines are cramped and poor |
Text: Do you know?' 'Lake? Oh! I really can't tell; but he'll soon tire of country life.
| Prompt Speaker | Ground Truth | Training Steps | F5-TTS + A-DMA (Ours) |
F5-TTS |
|---|---|---|---|---|
|
do you know lake oh i really cant tell but hell soon tire of country life |
100k |
do you know (missing) oh i really cant tell but hell soon tire of country life |
do you know (missing) i really cant tell but hell soon tire of country life |
|
|
200k |
do you know (missing) oh i really cant tell but hell soon tire of country life |
do you know (missing) i really cant tell but hell soon tire of country life |
||
|
300k |
do you know lake oh i really cant tell but hell soon tire of country life |
do you know lake (missing) i really cant tell but hell soon tire of country life |
||
|
400k |
do you know lake oh i really cant tell but hell soon tire of country life |
do you know (missing) oh i really cant tell but hell soon tire of country life |
Text: As regards my robbing the company, I'll say that I saved them a heavy loss one day.
| Prompt Speaker | Ground Truth | Training Steps | F5-TTS + A-DMA (Ours) |
F5-TTS |
|---|---|---|---|---|
|
as regards my robbing the company ill say that i saved him a heavy loss one day |
100k |
as regards my robbing the company ill say that i saved them a heavy loss one day |
as regarding (missing) robbing the company ill say they (missing) saved them a heavy loss one day |
|
|
200k |
as regards my robbing the company ill say that i saved them a heavy loss one day |
as regards my robbing the company ill say that i saved them a heavy loss one day |
||
|
300k |
as regards my robbing the company ill say that i saved them a heavy loss one day |
as regards my robbing the company ill say that i saved them a heavy loss one day |
||
|
400k |
as regards my robbing the company ill say that i saved them a heavy loss one day |
as regards my robbing the company ill say that i saved them a heavy loss one day |
Model Comparison
* Each text below each audio is ASR transcribed and normalized.
Text: And to think we can save all that misery and despair by the payment of a hundred and fifty dollars!
| Prompt Speaker | Ground Truth | F5-TTS + A-DMA (Ours) |
F5-TTS | E2-TTS + A-DMA (Ours) |
E2-TTS | USLM |
|---|---|---|---|---|---|---|
|
and to think we can save all that misery and despair by the payment of a hundred and fifty dollars |
and to think we can save all that misery and despair by the payment of a hundred and fifty dollars |
and to think we can save all that miserisery and despair by the payment of a hundred and fifty dollars |
and to think we can save all that misery and despair by the payment of a hundred and fifty dollars |
and to think we can save all that misery and despair by the payment of a hundred and fifty dollars |
and to think we can save all that misery and despair by the payment of a hundred and fifty dollars |
Text: Why, it's Goliath as usual! they both cried, peering in.
| Prompt Speaker | Ground Truth | F5-TTS + A-DMA (Ours) |
F5-TTS | E2-TTS + A-DMA (Ours) |
E2-TTS | USLM |
|---|---|---|---|---|---|---|
|
why its goliath as usual they both cried peering in |
why its goliath as usual they both cried peering in |
why its goliath as usual they both cried peering in |
why its goliath as usual they both cried peering in |
why its goliath as you will they both cried peering in |
why its goliaths jewel (missing) they both cried peering in |
Text: The first of our vague but indubitable data is that there is knowledge of the past.
| Prompt Speaker | Ground Truth | F5-TTS + A-DMA (Ours) |
F5-TTS | E2-TTS + A-DMA (Ours) |
E2-TTS | USLM |
|---|---|---|---|---|---|---|
|
the first of our vague but indubitable data is that there is knowledge of the past |
the first of our vague but indubitable data is that there is knowledge of the past |
the first of our vague but indubitable data is that there is knowledge of the past |
the first of our vague but indubitable data is that there is knowledge of the past |
the first of our vague but indubitable data is that there is knowledge of the past |
the first of our vague but indubitable data is that there is knowledge of the past |
Text: It has no beauty whatsoever, no specialty of picturesqueness; and all its lines are cramped and poor.
| Prompt Speaker | Ground Truth | F5-TTS + A-DMA (Ours) |
F5-TTS | E2-TTS + A-DMA (Ours) |
E2-TTS | USLM |
|---|---|---|---|---|---|---|
|
it has no beauty whatsoever no specialty of picturesqueness and all its lines are cramped and poor |
it has no beauty whatsoever no specialty of picturesqueness and all its lines are cramped and poor |
it has no beauty whatsoever no specie of picturesqueness and all its lines are cramped and poor |
it has no beauty whatsoever no specialty of picturesqueness and all its lines are cramped and poor |
it has no beauty whatsoever no specialty of picturesqueness and all its lines are cramped and poor |
it has no beauty whatsoever no specially of picturesqueness and all its lines are cramped and poor |
Text: Do you know?' 'Lake? Oh! I really can't tell; but he'll soon tire of country life.
| Prompt Speaker | Ground Truth | F5-TTS + A-DMA (Ours) |
F5-TTS | E2-TTS + A-DMA (Ours) |
E2-TTS | USLM |
|---|---|---|---|---|---|---|
|
do you know lake oh i really cant tell but hell soon tire of country life |
do you know lake oh i really cant tell but hell soon tire of country life |
do you know (missing) oh i really cant tell but hell soon tire of country life |
do you know lake oh i really cant tell but hell soon tire of country life |
do you know lake oh i really cant tell but hell soon tire of country life |
do you know lake oi (missing) really cant tell but hell soon tire of country life |
Text: As regards my robbing the company, I'll say that I saved them a heavy loss one day.
| Prompt Speaker | Ground Truth | F5-TTS + A-DMA (Ours) |
F5-TTS | E2-TTS + A-DMA (Ours) |
E2-TTS | USLM |
|---|---|---|---|---|---|---|
|
as regards my robbing the company ill say that i saved him a heavy loss one day |
as regards my robbing the company ill say that i saved them a heavy loss one day |
as regards my robbing the company ill say that i saved them a heavy loss one day |
as regards my robbing the company ill say that i saved them a heavy loss one day |
as regards my robbing the company ill say that i saved them a heavy loss one day |
as you gods my robin the company ill say that i saved them a heavy loss one day |