  Machine learning (ML) models have been shown to leak private information from
their training datasets. Differential Privacy (DP), typically implemented
through the differential private stochastic gradient descent algorithm
(DP-SGD), has become the standard solution to bound leakage from the models.
Despite recent improvements, DP-SGD-based approaches for private learning still
usually struggle in the high privacy ($\varepsilon\le1)$ and low data regimes,
and when the private training datasets are imbalanced. To overcome these
limitations, we propose Differentially Private Prototype Learning (DPPL) as a
new paradigm for private transfer learning. DPPL leverages publicly pre-trained
encoders to extract features from private data and generates DP prototypes that
represent each private class in the embedding space and can be publicly
released for inference. Since our DP prototypes can be obtained from only a few
private training data points and without iterative noise addition, they offer
high-utility predictions and strong privacy guarantees even under the notion of
pure DP. We additionally show that privacy-utility trade-offs can be further
improved when leveraging the public data beyond pre-training of the encoder: in
particular, we can privately sample our DP prototypes from the publicly
available data points used to train the encoder. Our experimental evaluation
with four state-of-the-art encoders, four vision datasets, and under different
data and imbalancedness regimes demonstrate DPPL's high performance under
strong privacy guarantees in challenging private learning setups.
