  The iterative nature of the expectation maximization (EM) algorithm presents
a challenge for privacy-preserving estimation, as each iteration increases the
amount of noise needed. We propose a practical private EM algorithm that
overcomes this challenge using two innovations: (1) a novel moment perturbation
formulation for differentially private EM (DP-EM), and (2) the use of two
recently developed composition methods to bound the privacy "cost" of multiple
EM iterations: the moments accountant (MA) and zero-mean concentrated
differential privacy (zCDP). Both MA and zCDP bound the moment generating
function of the privacy loss random variable and achieve a refined tail bound,
which effectively decrease the amount of additive noise. We present empirical
results showing the benefits of our approach, as well as similar performance
between these two composition methods in the DP-EM setting for Gaussian mixture
models. Our approach can be readily extended to many iterative learning
algorithms, opening up various exciting future directions.
