  Nowadays, owners and developers of deep learning models must consider
stringent privacy-preservation rules of their training data, usually
crowd-sourced and retaining sensitive information. The most widely adopted
method to enforce privacy guarantees of a deep learning model nowadays relies
on optimization techniques enforcing differential privacy. According to the
literature, this approach has proven to be a successful defence against several
models' privacy attacks, but its downside is a substantial degradation of the
models' performance. In this work, we compare the effectiveness of the
differentially-private stochastic gradient descent (DP-SGD) algorithm against
standard optimization practices with regularization techniques. We analyze the
resulting models' utility, training performance, and the effectiveness of
membership inference and model inversion attacks against the learned models.
Finally, we discuss differential privacy's flaws and limits and empirically
demonstrate the often superior privacy-preserving properties of dropout and
l2-regularization.
