  Differentially private training offers a protection which is usually
interpreted as a guarantee against membership inference attacks. By proxy, this
guarantee extends to other threats like reconstruction attacks attempting to
extract complete training examples. Recent works provide evidence that if one
does not need to protect against membership attacks but instead only wants to
protect against training data reconstruction, then utility of private models
can be improved because less noise is required to protect against these more
ambitious attacks. We investigate this further in the context of DP-SGD, a
standard algorithm for private deep learning, and provide an upper bound on the
success of any reconstruction attack against DP-SGD together with an attack
that empirically matches the predictions of our bound. Together, these two
results open the door to fine-grained investigations on how to set the privacy
parameters of DP-SGD in practice to protect against reconstruction attacks.
Finally, we use our methods to demonstrate that different settings of the
DP-SGD parameters leading to the same DP guarantees can result in significantly
different success rates for reconstruction, indicating that the DP guarantee
alone might not be a good proxy for controlling the protection against
reconstruction attacks.
