  Score-based modeling through stochastic differential equations (SDEs) has
provided a new perspective on diffusion models, and demonstrated superior
performance on continuous data. However, the gradient of the log-likelihood
function, i.e., the score function, is not properly defined for discrete
spaces. This makes it non-trivial to adapt \textcolor{\cdiff}{the score-based
modeling} to categorical data. In this paper, we extend diffusion models to
discrete variables by introducing a stochastic jump process where the reverse
process denoises via a continuous-time Markov chain. This formulation admits an
analytical simulation during backward sampling. To learn the reverse process,
we extend score matching to general categorical data and show that an unbiased
estimator can be obtained via simple matching of the conditional marginal
distributions. We demonstrate the effectiveness of the proposed method on a set
of synthetic and real-world music and image benchmarks.
