  Large-scale diffusion models have shown outstanding generative abilities
across multiple modalities including images, videos, and audio. However,
text-to-speech (TTS) systems typically involve domain-specific modeling factors
(e.g., phonemes and phoneme-level durations) to ensure precise temporal
alignments between text and speech, which hinders the efficiency and
scalability of diffusion models for TTS. In this work, we present an efficient
and scalable Diffusion Transformer (DiT) that utilizes off-the-shelf
pre-trained text and speech encoders. Our approach addresses the challenge of
text-speech alignment via cross-attention mechanisms with the prediction of the
total length of speech representations. To achieve this, we enhance the DiT
architecture to suit TTS and improve the alignment by incorporating semantic
guidance into the latent space of speech. We scale the training dataset and the
model size to 82K hours and 790M parameters, respectively. Our extensive
experiments demonstrate that the large-scale diffusion model for TTS without
domain-specific modeling not only simplifies the training pipeline but also
yields superior or comparable zero-shot performance to state-of-the-art TTS
models in terms of naturalness, intelligibility, and speaker similarity. Our
speech samples are available at https://ditto-tts.github.io.
