  Multiple medical institutions collaboratively training a model using
federated learning (FL) has become a promising solution for maximizing the
potential of data-driven models, yet the non-independent and identically
distributed (non-iid) data in medical images is still an outstanding challenge
in real-world practice. The feature heterogeneity caused by diverse scanners or
protocols introduces a drift in the learning process, in both local (client)
and global (server) optimizations, which harms the convergence as well as model
performance. Many previous works have attempted to address the non-iid issue by
tackling the drift locally or globally, but how to jointly solve the two
essentially coupled drifts is still unclear. In this work, we concentrate on
handling both local and global drifts and introduce a new harmonizing framework
called HarmoFL. First, we propose to mitigate the local update drift by
normalizing amplitudes of images transformed into the frequency domain to mimic
a unified imaging setting, in order to generate a harmonized feature space
across local clients. Second, based on harmonized features, we design a client
weight perturbation guiding each local model to reach a flat optimum, where a
neighborhood area of the local optimal solution has a uniformly low loss.
Without any extra communication cost, the perturbation assists the global model
to optimize towards a converged optimal solution by aggregating several local
flat optima. We have theoretically analyzed the proposed method and empirically
conducted extensive experiments on three medical image classification and
segmentation tasks, showing that HarmoFL outperforms a set of recent
state-of-the-art methods with promising convergence behavior. Code is available
at https://github.com/med-air/HarmoFL.
