  In the domains of image and audio, diffusion models have shown impressive
performance. However, their application to discrete data types, such as
language, has often been suboptimal compared to autoregressive generative
models. This paper tackles the challenge of improving discrete diffusion models
by introducing a structured forward process that leverages the inherent
information hierarchy in discrete categories, such as words in text. Our
approach biases the generative process to produce certain categories before
others, resulting in a notable improvement in log-likelihood scores on the
text8 dataset. This work paves the way for more advances in discrete diffusion
models with potentially significant enhancements in performance.
