  This survey paper provides a comprehensive review of the use of diffusion
models in natural language processing (NLP). Diffusion models are a class of
mathematical models that aim to capture the diffusion of information or signals
across a network or manifold. In NLP, diffusion models have been used in a
variety of applications, such as natural language generation, sentiment
analysis, topic modeling, and machine translation. This paper discusses the
different formulations of diffusion models used in NLP, their strengths and
limitations, and their applications. We also perform a thorough comparison
between diffusion models and alternative generative models, specifically
highlighting the autoregressive (AR) models, while also examining how diverse
architectures incorporate the Transformer in conjunction with diffusion models.
Compared to AR models, diffusion models have significant advantages for
parallel generation, text interpolation, token-level controls such as syntactic
structures and semantic contents, and robustness. Exploring further
permutations of integrating Transformers into diffusion models would be a
valuable pursuit. Also, the development of multimodal diffusion models and
large-scale diffusion language models with notable capabilities for few-shot
learning would be important directions for the future advance of diffusion
models in NLP.
