  Exploiting gradient leakage to reconstruct supposedly private training data,
gradient inversion attacks are an ubiquitous threat in collaborative learning
of neural networks. To prevent gradient leakage without suffering from severe
loss in model performance, recent work proposed a PRivacy EnhanCing mODulE
(PRECODE) based on variational modeling as extension for arbitrary model
architectures. In this work, we investigate the effect of PRECODE on gradient
inversion attacks to reveal its underlying working principle. We show that
variational modeling induces stochasticity on PRECODE's and its subsequent
layers' gradients that prevents gradient attacks from convergence. By
purposefully omitting those stochastic gradients during attack optimization, we
formulate an attack that can disable PRECODE's privacy preserving effects. To
ensure privacy preservation against such targeted attacks, we propose PRECODE
with Partial Perturbation (PPP), as strategic combination of variational
modeling and partial gradient perturbation. We conduct an extensive empirical
study on four seminal model architectures and two image classification
datasets. We find all architectures to be prone to gradient leakage, which can
be prevented by PPP. In result, we show that our approach requires less
gradient perturbation to effectively preserve privacy without harming model
performance.
