  Denoising Diffusion Probabilistic Models have shown extraordinary ability on
various generative tasks. However, their slow inference speed renders them
impractical in speech synthesis. This paper proposes a linear diffusion model
(LinDiff) based on an ordinary differential equation to simultaneously reach
fast inference and high sample quality. Firstly, we employ linear interpolation
between the target and noise to design a diffusion sequence for training, while
previously the diffusion path that links the noise and target is a curved
segment. When decreasing the number of sampling steps (i.e., the number of line
segments used to fit the path), the ease of fitting straight lines compared to
curves allows us to generate higher quality samples from a random noise with
fewer iterations. Secondly, to reduce computational complexity and achieve
effective global modeling of noisy speech, LinDiff employs a patch-based
processing approach that partitions the input signal into small patches. The
patch-wise token leverages Transformer architecture for effective modeling of
global information. Adversarial training is used to further improve the sample
quality with decreased sampling steps. We test proposed method with speech
synthesis conditioned on acoustic feature (Mel-spectrograms). Experimental
results verify that our model can synthesize high-quality speech even with only
one diffusion step. Both subjective and objective evaluations demonstrate that
our model can synthesize speech of a quality comparable to that of
autoregressive models with faster synthesis speed (3 diffusion steps).
