  Diffusion models have quickly become the go-to paradigm for generative
modelling of perceptual signals (such as images and sound) through iterative
refinement. Their success hinges on the fact that the underlying physical
phenomena are continuous. For inherently discrete and categorical data such as
language, various diffusion-inspired alternatives have been proposed. However,
the continuous nature of diffusion models conveys many benefits, and in this
work we endeavour to preserve it. We propose CDCD, a framework for modelling
categorical data with diffusion models that are continuous both in time and
input space. We demonstrate its efficacy on several language modelling tasks.
