  Despite imperfect score-matching causing drift in training and sampling
distributions of diffusion models, recent advances in diffusion-based acoustic
models have revolutionized data-sufficient single-speaker Text-to-Speech (TTS)
approaches, with Grad-TTS being a prime example. However, the sampling drift
problem leads to these approaches struggling in multi-speaker scenarios in
practice due to more complex target data distribution compared to
single-speaker scenarios. In this paper, we present Multi-GradSpeech, a
multi-speaker diffusion-based acoustic models which introduces the Consistent
Diffusion Model (CDM) as a generative modeling approach. We enforce the
consistency property of CDM during the training process to alleviate the
sampling drift problem in the inference stage, resulting in significant
improvements in multi-speaker TTS performance. Our experimental results
corroborate that our proposed approach can improve the performance of different
speakers involved in multi-speaker TTS compared to Grad-TTS, even outperforming
the fine-tuning approach. Audio samples are available at
https://welkinyang.github.io/multi-gradspeech/
