  Protecting privacy in learning while maintaining the model performance has
become increasingly critical in many applications that involve sensitive data.
Private Gradient Descent (PGD) is a commonly used private learning framework,
which noises gradients based on the Differential Privacy protocol. Recent
studies show that \emph{dynamic privacy schedules} of decreasing noise
magnitudes can improve loss at the final iteration, and yet theoretical
understandings of the effectiveness of such schedules and their connections to
optimization algorithms remain limited. In this paper, we provide comprehensive
analysis of noise influence in dynamic privacy schedules to answer these
critical questions. We first present a dynamic noise schedule minimizing the
utility upper bound of PGD, and show how the noise influence from each
optimization step collectively impacts utility of the final model. Our study
also reveals how impacts from dynamic noise influence change when momentum is
used. We empirically show the connection exists for general non-convex losses,
and the influence is greatly impacted by the loss curvature.
