  Federated Learning (FL) enables the multiple participating devices to
collaboratively contribute to a global neural network model while keeping the
training data locally. Unlike the centralized training setting, the non-IID and
imbalanced (statistical heterogeneity) training data of FL is distributed in
the federated network, which will increase the divergences between the local
models and global model, further degrading performance. In this paper, we
propose a novel clustered federated learning (CFL) framework FedGroup, in which
we 1) group the training of clients based on the similarities between the
clients' optimization directions for high training performance; 2) construct a
new data-driven distance measure to improve the efficiency of the client
clustering procedure. 3) implement a newcomer device cold start mechanism based
on the auxiliary global model for framework scalability and practicality.
  FedGroup can achieve improvements by dividing joint optimization into groups
of sub-optimization and can be combined with FL optimizer FedProx. The
convergence and complexity are analyzed to demonstrate the efficiency of our
proposed framework. We also evaluate FedGroup and FedGrouProx (combined with
FedProx) on several open datasets and made comparisons with related CFL
frameworks. The results show that FedGroup can significantly improve absolute
test accuracy by +14.1% on FEMNIST compared to FedAvg. +3.4% on Sentiment140
compared to FedProx, +6.9% on MNIST compared to FeSEM.
