  Many applications of machine learning, for example in health care, would
benefit from methods that can guarantee privacy of data subjects. Differential
privacy (DP) has become established as a standard for protecting learning
results. The standard DP algorithms require a single trusted party to have
access to the entire data, which is a clear weakness. We consider DP Bayesian
learning in a distributed setting, where each party only holds a single sample
or a few samples of the data. We propose a learning strategy based on a secure
multi-party sum function for aggregating summaries from data holders and the
Gaussian mechanism for DP. Our method builds on an asymptotically optimal and
practically efficient DP Bayesian inference with rapidly diminishing extra
cost.
