  We present an approach for generating differentially private synthetic text
using large language models (LLMs), via private prediction. In the private
prediction framework, we only require the output synthetic data to satisfy
differential privacy guarantees. This is in contrast to approaches that train a
generative model on potentially sensitive user-supplied source data and seek to
ensure the model itself is safe to release.
  We prompt a pretrained LLM with source data, but ensure that next-token
predictions are made with differential privacy guarantees. Previous work in
this paradigm reported generating a small number of examples (<10) at
reasonable privacy levels, an amount of data that is useful only for downstream
in-context learning or prompting. In contrast, we make changes that allow us to
generate thousands of high-quality synthetic data points, greatly expanding the
set of potential applications. Our improvements come from an improved privacy
analysis and a better private selection mechanism, which makes use of the
equivalence between the softmax layer for sampling tokens in LLMs and the
exponential mechanism. Furthermore, we introduce a novel use of public
predictions via the sparse vector technique, in which we do not pay privacy
costs for tokens that are predictable without sensitive data; we find this to
be particularly effective for structured data.
