  We introduce AdaMix, an adaptive differentially private algorithm for
training deep neural network classifiers using both private and public image
data. While pre-training language models on large public datasets has enabled
strong differential privacy (DP) guarantees with minor loss of accuracy, a
similar practice yields punishing trade-offs in vision tasks. A few-shot or
even zero-shot learning baseline that ignores private data can outperform
fine-tuning on a large private dataset. AdaMix incorporates few-shot training,
or cross-modal zero-shot learning, on public data prior to private fine-tuning,
to improve the trade-off. AdaMix reduces the error increase from the
non-private upper bound from the 167-311\% of the baseline, on average across 6
datasets, to 68-92\% depending on the desired privacy level selected by the
user. AdaMix tackles the trade-off arising in visual classification, whereby
the most privacy sensitive data, corresponding to isolated points in
representation space, are also critical for high classification accuracy. In
addition, AdaMix comes with strong theoretical privacy guarantees and
convergence analysis.
