  Text prediction models, when used in applications like email clients or word
processors, must protect user data privacy and adhere to model size
constraints. These constraints are crucial to meet memory and inference time
requirements, as well as to reduce inference costs. Building small, fast, and
private domain-specific language models is a thriving area of research. In this
work, we show that a careful pre-training on a \emph{subset} of the public
dataset that is guided by the private dataset is crucial to train small
language models with differential privacy. On standard benchmarks, small models
trained with our new framework achieve state-of-the-art performance. In
addition to performance improvements, our results demonstrate that smaller
models, through careful pre-training and private fine-tuning, can match the
performance of much larger models that do not have access to private data. This
underscores the potential of private learning for model compression and
enhanced efficiency.
