  Differentially private stochastic gradient descent (DP-SGD) is the standard
algorithm for training machine learning models under differential privacy (DP).
The major drawback of DP-SGD is the drop in utility which prior work has
comprehensively studied. However, in practice another major drawback that
hinders the large-scale deployment is the significantly higher computational
cost. We conduct a comprehensive empirical study to quantify the computational
cost of training deep learning models under DP and benchmark methods that aim
at reducing the cost. Among these are more efficient implementations of DP-SGD
and training with lower precision. Finally, we study the scaling behaviour
using up to 80 GPUs.
