  In the privacy-utility tradeoff of a model trained on benchmark language and
vision tasks, remarkable improvements have been widely reported with the use of
pretraining on publicly available data. This is in part due to the benefits of
transfer learning, which is the standard motivation for pretraining in
non-private settings. However, the stark contrast in the improvement achieved
through pretraining under privacy compared to non-private settings suggests
that there may be a deeper, distinct cause driving these gains. To explain this
phenomenon, we hypothesize that the non-convex loss landscape of a model
training necessitates an optimization algorithm to go through two phases. In
the first, the algorithm needs to select a good "basin" in the loss landscape.
In the second, the algorithm solves an easy optimization within that basin. The
former is a harder problem to solve with private data, while the latter is
harder to solve with public data due to a distribution shift or data scarcity.
Guided by this intuition, we provide theoretical constructions that provably
demonstrate the separation between private training with and without public
pretraining. Further, systematic experiments on CIFAR10 and LibriSpeech provide
supporting evidence for our hypothesis.
