  Machine learning models can be trained with formal privacy guarantees via
differentially private optimizers such as DP-SGD. In this work, we study such
privacy guarantees when the adversary only accesses the final model, i.e.,
intermediate model updates are not released. In the existing literature, this
hidden state threat model exhibits a significant gap between the lower bound
provided by empirical privacy auditing and the theoretical upper bound provided
by privacy accounting. To challenge this gap, we propose to audit this threat
model with adversaries that craft a gradient sequence to maximize the privacy
loss of the final model without accessing intermediate models. We demonstrate
experimentally how this approach consistently outperforms prior attempts at
auditing the hidden state model. When the crafted gradient is inserted at every
optimization step, our results imply that releasing only the final model does
not amplify privacy, providing a novel negative result. On the other hand, when
the crafted gradient is not inserted at every step, we show strong evidence
that a privacy amplification phenomenon emerges in the general non-convex
setting (albeit weaker than in convex regimes), suggesting that existing
privacy upper bounds can be improved.
