  Differentially Private Stochastic Gradient Descent (DP-SGD) and its variants
have been proposed to ensure rigorous privacy for fine-tuning large-scale
pre-trained language models. However, they rely heavily on the Gaussian
mechanism, which may overly perturb the gradients and degrade the accuracy,
especially in stronger privacy regimes (e.g., the privacy budget $\epsilon <
3$). To address such limitations, we propose a novel Language Model-based
Optimal Differential Privacy (LMO-DP) mechanism, which takes the first step to
enable the tight composition of accurately fine-tuning (large) language models
with a sub-optimal DP mechanism, even in strong privacy regimes (e.g., $0.1\leq
\epsilon<3$). Furthermore, we propose a novel offline optimal noise search
method to efficiently derive the sub-optimal DP that significantly reduces the
noise magnitude. For instance, fine-tuning RoBERTa-large (with 300M parameters)
on the SST-2 dataset can achieve an accuracy of 92.20% (given $\epsilon=0.3$,
$\delta=10^{-10}$) by drastically outperforming the Gaussian mechanism (e.g.,
$\sim 50\%$ for small $\epsilon$ and $\delta$). We also draw similar findings
on the text generation tasks on GPT-2. Finally, to our best knowledge, LMO-DP
is also the first solution to accurately fine-tune Llama-2 with strong
differential privacy guarantees. The code will be released soon and available
upon request.
