  We present a novel method for training score-based generative models which
uses nonlinear noising dynamics to improve learning of structured
distributions. Generalizing to a nonlinear drift allows for additional
structure to be incorporated into the dynamics, thus making the training better
adapted to the data, e.g., in the case of multimodality or (approximate)
symmetries. Such structure can be obtained from the data by an inexpensive
preprocessing step. The nonlinear dynamics introduces new challenges into
training which we address in two ways: 1) we develop a new nonlinear denoising
score matching (NDSM) method, 2) we introduce neural control variates in order
to reduce the variance of the NDSM training objective. We demonstrate the
effectiveness of this method on several examples: a) a collection of
low-dimensional examples, motivated by clustering in latent space, b)
high-dimensional images, addressing issues with mode collapse, small training
sets, and approximate symmetries, the latter being a challenge for methods
based on equivariant neural networks, which require exact symmetries.
