  Federated learning enables edge devices to train a global model
collaboratively without exposing their data. Despite achieving outstanding
advantages in computing efficiency and privacy protection, federated learning
faces a significant challenge when dealing with non-IID data, i.e., data
generated by clients that are typically not independent and identically
distributed. In this paper, we tackle a new type of Non-IID data, called
cluster-skewed non-IID, discovered in actual data sets. The cluster-skewed
non-IID is a phenomenon in which clients can be grouped into clusters with
similar data distributions. By performing an in-depth analysis of the behavior
of a classification model's penultimate layer, we introduce a metric that
quantifies the similarity between two clients' data distributions without
violating their privacy. We then propose an aggregation scheme that guarantees
equality between clusters. In addition, we offer a novel local training
regularization based on the knowledge-distillation technique that reduces the
overfitting problem at clients and dramatically boosts the training scheme's
performance. We theoretically prove the superiority of the proposed aggregation
over the benchmark FedAvg. Extensive experimental results on both standard
public datasets and our in-house real-world dataset demonstrate that the
proposed approach improves accuracy by up to 16% compared to the FedAvg
algorithm.
