  We introduce a multiple target optimization framework for DP-SGD referred to
as pro-active DP. In contrast to traditional DP accountants, which are used to
track the expenditure of privacy budgets, the pro-active DP scheme allows one
to a-priori select parameters of DP-SGD based on a fixed privacy budget (in
terms of $\epsilon$ and $\delta$) in such a way to optimize the anticipated
utility (test accuracy) the most. To achieve this objective, we first propose
significant improvements to the moment account method, presenting a closed-form
$(\epsilon,\delta)$-DP guarantee that connects all parameters in the DP-SGD
setup. We show that DP-SGD is $(\epsilon<0.5,\delta=1/N)$-DP if
$\sigma=\sqrt{2(\epsilon +\ln(1/\delta))/\epsilon}$ with $T$ at least $\approx
2k^2/\epsilon$ and $(2/e)^2k^2-1/2\geq \ln(N)$, where $T$ is the total number
of rounds, and $K=kN$ is the total number of gradient computations where $k$
measures $K$ in number of epochs of size $N$ of the local data set. We prove
that our expression is close to tight in that if $T$ is more than a constant
factor $\approx 4$ smaller than the lower bound $\approx 2k^2/\epsilon$, then
the $(\epsilon,\delta)$-DP guarantee is violated. The above DP guarantee can be
enhanced in thatDP-SGD is $(\epsilon, \delta)$-DP if $\sigma =
\sqrt{2(\epsilon+\ln(1/\delta))/\epsilon}$ with $T$ at least $\approx
2k^2/\epsilon$ together with two additional, less intuitive, conditions that
allow larger $\epsilon\geq 0.5$. Our DP theory allows us to create a utility
graph and DP calculator. These tools link privacy and utility objectives and
search for optimal experiment setups, efficiently taking into account both
accuracy and privacy objectives, as well as implementation goals. We furnish a
comprehensive implementation flow of our proactive DP, with rigorous
experiments to showcase the proof-of-concept.
