  We observe that the traditional use of DP with the Adam optimizer introduces
a bias in the second moment estimation, due to the addition of independent
noise in the gradient computation. This bias leads to a different scaling for
low variance parameter updates, that is inconsistent with the behavior of
non-private Adam, and Adam's sign descent interpretation. Empirically,
correcting the bias introduced by DP noise significantly improves the
optimization performance of DP-Adam.
