  Differential privacy guarantees allow the results of a statistical analysis
involving sensitive data to be released without compromising the privacy of any
individual taking part. Achieving such guarantees generally requires the
injection of noise, either directly into parameter estimates or into the
estimation process. Instead of artificially introducing perturbations, sampling
from Bayesian posterior distributions has been shown to be a special case of
the exponential mechanism, producing consistent, and efficient private
estimates without altering the data generative process. The application of
current approaches has, however, been limited by their strong bounding
assumptions which do not hold for basic models, such as simple linear
regressors. To ameliorate this, we propose $\beta$D-Bayes, a posterior sampling
scheme from a generalised posterior targeting the minimisation of the
$\beta$-divergence between the model and the data generating process. This
provides private estimation that is generally applicable without requiring
changes to the underlying model and consistently learns the data generating
parameter. We show that $\beta$D-Bayes produces more precise inference
estimation for the same privacy guarantees, and further facilitates
differentially private estimation via posterior sampling for complex
classifiers and continuous regression models such as neural networks for the
first time.
