  Federated learning (FL) supports distributed training of a global machine
learning model across multiple devices with the help of a central server.
However, data heterogeneity across different devices leads to the client model
drift issue and results in model performance degradation and poor model
fairness. To address the issue, we design Federated learning with global-local
Knowledge Fusion (FedKF) scheme in this paper. The key idea in FedKF is to let
the server return the global knowledge to be fused with the local knowledge in
each training round so that the local model can be regularized towards the
global optima. Therefore, the client model drift issue can be mitigated. In
FedKF, we first propose the active-inactive model aggregation technique that
supports a precise global knowledge representation. Then, we propose a
data-free knowledge distillation (KD) approach to enable each client model to
learn the global knowledge (embedded in the global model) while each client
model can still learn the local knowledge (embedded in the local dataset)
simultaneously, thereby realizing the global-local knowledge fusion process.
The theoretical analysis and intensive experiments demonstrate the superiority
of FedKF over previous solutions.
