  Many machine learning applications are based on data collected from people,
such as their tastes and behaviour as well as biological traits and genetic
data. Regardless of how important the application might be, one has to make
sure individuals' identities or the privacy of the data are not compromised in
the analysis. Differential privacy constitutes a powerful framework that
prevents breaching of data subject privacy from the output of a computation.
Differentially private versions of many important Bayesian inference methods
have been proposed, but there is a lack of an efficient unified approach
applicable to arbitrary models. In this contribution, we propose a
differentially private variational inference method with a very wide
applicability. It is built on top of doubly stochastic variational inference, a
recent advance which provides a variational solution to a large class of
models. We add differential privacy into doubly stochastic variational
inference by clipping and perturbing the gradients. The algorithm is made more
efficient through privacy amplification from subsampling. We demonstrate the
method can reach an accuracy close to non-private level under reasonably strong
privacy guarantees, clearly improving over previous sampling-based alternatives
especially in the strong privacy regime.
