  Differentially private (DP) release of multidimensional statistics typically
considers an aggregate sensitivity, e.g. the vector norm of a high-dimensional
vector. However, different dimensions of that vector might have widely
different magnitudes and therefore DP perturbation disproportionately affects
the signal across dimensions. We observe this problem in the gradient release
of the DP-SGD algorithm when using it for variational inference (VI), where it
manifests in poor convergence as well as high variance in outputs for certain
variational parameters, and make the following contributions: (i) We
mathematically isolate the cause for the difference in magnitudes between
gradient parts corresponding to different variational parameters. Using this as
prior knowledge we establish a link between the gradients of the variational
parameters, and propose an efficient while simple fix for the problem to obtain
a less noisy gradient estimator, which we call $\textit{aligned}$ gradients.
This approach allows us to obtain the updates for the covariance parameter of a
Gaussian posterior approximation without a privacy cost. We compare this to
alternative approaches for scaling the gradients using analytically derived
preconditioning, e.g. natural gradients. (ii) We suggest using iterate
averaging over the DP parameter traces recovered during the training, to reduce
the DP-induced noise in parameter estimates at no additional cost in privacy.
Finally, (iii) to accurately capture the additional uncertainty DP introduces
to the model parameters, we infer the DP-induced noise from the parameter
traces and include that in the learned posteriors to make them $\textit{noise
aware}$. We demonstrate the efficacy of our proposed improvements through
various experiments on real data.
