  Differentially private stochastic gradient descent (DP-SGD) adds noise to
gradients in back-propagation, safeguarding training data from privacy leakage,
particularly membership inference. It fails to cover (inference-time) threats
like embedding inversion and sensitive attribute inference. It is also costly
in storage and computation when used to fine-tune large pre-trained language
models (LMs).
  We propose DP-Forward, which directly perturbs embedding matrices in the
forward pass of LMs. It satisfies stringent local DP requirements for training
and inference data. To instantiate it using the smallest matrix-valued noise,
we devise an analytic matrix Gaussian~mechanism (aMGM) by drawing possibly
non-i.i.d. noise from a matrix Gaussian distribution. We then investigate
perturbing outputs from different hidden (sub-)layers of LMs with aMGM noises.
Its utility on three typical tasks almost hits the non-private baseline and
outperforms DP-SGD by up to 7.7pp at a moderate privacy level. It saves
3$\times$ time and memory costs compared to DP-SGD with the latest high-speed
library. It also reduces the average success rates of embedding inversion and
sensitive attribute inference by up to 88pp and 41pp, respectively, whereas
DP-SGD fails.
