  In this paper, we revisit the problem of using in-distribution public data to
improve the privacy/utility trade-offs for differentially private (DP) model
training. (Here, public data refers to auxiliary data sets that have no privacy
concerns.) We design a natural variant of DP mirror descent, where the DP
gradients of the private/sensitive data act as the linear term, and the loss
generated by the public data as the mirror map.
  We show that, for linear regression with feature vectors drawn from a
non-isotropic sub-Gaussian distribution, our algorithm, PDA-DPMD (a variant of
mirror descent), provides population risk guarantees that are asymptotically
better than the best known guarantees under DP (without having access to public
data), when the number of public data samples ($n_{\sf pub}$) is sufficiently
large. We further show that our algorithm has natural "noise stability"
properties that control the variance due to noise added to ensure DP.
  We demonstrate the efficacy of our algorithm by showing privacy/utility
trade-offs on four benchmark datasets (StackOverflow, WikiText-2, CIFAR-10, and
EMNIST). We show that our algorithm not only significantly improves over
traditional DP-SGD, which does not have access to public data, but to our
knowledge is the first to improve over DP-SGD on models that have been
pre-trained with public data.
