  Federated training of large deep neural networks can often be restrictive due
to the increasing costs of communicating the updates with increasing model
sizes. Various model pruning techniques have been designed in centralized
settings to reduce inference times. Combining centralized pruning techniques
with federated training seems intuitive for reducing communication costs -- by
pruning the model parameters right before the communication step. Moreover,
such a progressive model pruning approach during training can also reduce
training times/costs. To this end, we propose FedSparsify, which performs model
pruning during federated training. In our experiments in centralized and
federated settings on the brain age prediction task (estimating a person's age
from their brain MRI), we demonstrate that models can be pruned up to 95%
sparsity without affecting performance even in challenging federated learning
environments with highly heterogeneous data distributions. One surprising
benefit of model pruning is improved model privacy. We demonstrate that models
with high sparsity are less susceptible to membership inference attacks, a type
of privacy attack.
