  Background: Federated learning methods offer the possibility of training
machine learning models on privacy-sensitive data sets, which cannot be easily
shared. Multiple regulations pose strict requirements on the storage and usage
of healthcare data, leading to data being in silos (i.e. locked-in at
healthcare facilities). The application of federated algorithms on these
datasets could accelerate disease diagnostic, drug development, as well as
improve patient care.
  Methods: We present an extensive evaluation of the impact of different
federation and differential privacy techniques when training models on the
open-source MIMIC-III dataset. We analyze a set of parameters influencing a
federated model performance, namely data distribution (homogeneous and
heterogeneous), communication strategies (communication rounds vs. local
training epochs), federation strategies (FedAvg vs. FedProx). Furthermore, we
assess and compare two differential privacy (DP) techniques during model
training: a stochastic gradient descent-based differential privacy algorithm
(DP-SGD), and a sparse vector differential privacy technique (DP-SVT).
  Results: Our experiments show that extreme data distributions across sites
(imbalance either in the number of patients or the positive label ratios
between sites) lead to a deterioration of model performance when trained using
the FedAvg strategy. This issue is resolved when using FedProx with the use of
appropriate hyperparameter tuning. Furthermore, the results show that both
differential privacy techniques can reach model performances similar to those
of models trained without DP, however at the expense of a large quantifiable
privacy leakage.
  Conclusions: We evaluate empirically the benefits of two federation
strategies and propose optimal strategies for the choice of parameters when
using differential privacy techniques.
