  Federated learning reduces the risk of information leakage, but remains
vulnerable to attacks. We investigate how several neural network design
decisions can defend against gradients inversion attacks. We show that
overlapping gradients provides numerical resistance to gradient inversion on
the highly vulnerable dense layer. Specifically, we propose to leverage
batching to maximise mixing of gradients by choosing an appropriate loss
function and drawing identical labels. We show that otherwise it is possible to
directly recover all vectors in a mini-batch without any numerical optimisation
due to the de-mixing nature of the cross entropy loss. To accurately assess
data recovery, we introduce an absolute variation distance (AVD) metric for
information leakage in images, derived from total variation. In contrast to
standard metrics, e.g. Mean Squared Error or Structural Similarity Index, AVD
offers a continuous metric for extracting information in noisy images. Finally,
our empirical results on information recovery from various inversion attacks
and training performance supports our defense strategies. These strategies are
also shown to be useful for deep convolutional neural networks such as LeNET
for image recognition. We hope that this study will help guide the development
of further strategies that achieve a trustful federation policy.
