  Differentially private (DP) optimization is the standard paradigm to learn
large neural networks that are accurate and privacy-preserving. The
computational cost for DP deep learning, however, is notoriously heavy due to
the per-sample gradient clipping. Existing DP implementations are 2-1000X more
costly in time and space complexity than the standard (non-private) training.
In this work, we develop a novel Book-Keeping (BK) technique that implements
existing DP optimizers (thus achieving the same accuracy), with a substantial
improvement on the computational cost. Specifically, BK enables DP training on
large models and high dimensional data to be roughly as fast and memory-saving
as the standard training, whereas previous DP algorithms can be inefficient or
incapable of training due to memory error. The computational advantage of BK is
supported by the complexity analysis as well as extensive experiments on vision
and language tasks. Our implementation achieves state-of-the-art (SOTA)
accuracy with very small extra cost: on GPT2 and at almost the same memory cost
(<1% overhead), BK has 1.03X the time complexity of the standard training
(0.83X training speed in practice), and 0.61X the time complexity of the most
efficient DP implementation (1.36X training speed in practice). We open-source
the codebase for the BK algorithm at the FastDP library
(https://github.com/awslabs/fast-differential-privacy).
