  We explore the training of generative adversarial networks with differential
privacy to anonymize image data sets. On MNIST, we numerically measure the
privacy-utility trade-off using parameters from $\epsilon$-$\delta$
differential privacy and the inception score. Our experiments uncover a
saturated training regime where an increasing privacy budget adds little to the
quality of generated images. We also explain analytically why differentially
private Adam optimization is independent of the gradient clipping parameter.
Furthermore, we highlight common errors in previous works on differentially
private deep learning, which we uncovered in recent literature. Throughout the
treatment of the subject, we hope to prevent erroneous estimates of anonymity
in the future.
