  The privacy preserving properties of Langevin dynamics with additive
isotropic noise have been extensively studied. However, the isotropic noise
assumption is very restrictive: (a) when adding noise to existing learning
algorithms to preserve privacy and maintain the best possible accuracy one
should take into account the relative magnitude of the outputs and their
correlations; (b) popular algorithms such as stochastic gradient descent (and
their continuous time limits) appear to possess anisotropic covariance
properties. To study the privacy risks for the anisotropic noise case, one
requires general results on the relative entropy between the laws of two
Stochastic Differential Equations with different drifts and diffusion
coefficients. Our main contribution is to establish such a bound using
stability estimates for solutions to the Fokker-Planck equations via functional
inequalities. With additional assumptions, the relative entropy bound implies
an $(\epsilon,\delta)$-differential privacy bound or translates to bounds on
the membership inference attack success and we show how anisotropic noise can
lead to better privacy-accuracy trade-offs. Finally, the benefits of
anisotropic noise are illustrated using numerical results in quadratic loss and
neural network setups.
