  In this study, we propose a simple and efficient Non-Autoregressive (NAR)
text-to-speech (TTS) system based on diffusion, named SimpleSpeech. Its
simpleness shows in three aspects: (1) It can be trained on the speech-only
dataset, without any alignment information; (2) It directly takes plain text as
input and generates speech through an NAR way; (3) It tries to model speech in
a finite and compact latent space, which alleviates the modeling difficulty of
diffusion. More specifically, we propose a novel speech codec model (SQ-Codec)
with scalar quantization, SQ-Codec effectively maps the complex speech signal
into a finite and compact latent space, named scalar latent space. Benefits
from SQ-Codec, we apply a novel transformer diffusion model in the scalar
latent space of SQ-Codec. We train SimpleSpeech on 4k hours of a speech-only
dataset, it shows natural prosody and voice cloning ability. Compared with
previous large-scale TTS models, it presents significant speech quality and
generation speed improvement. Demos are released.
