  Federated Learning (FL) has emerged as a promising solution to perform deep
learning on different data owners without exchanging raw data. However, non-IID
data has been a key challenge in FL, which could significantly degrade the
accuracy of the final model. Among different non-IID types, label skews have
been challenging and common in image classification and other tasks. Instead of
averaging the local models in most previous studies, we propose FedConcat, a
simple and effective approach that concatenates these local models as the base
of the global model to effectively aggregate the local knowledge. To reduce the
size of the global model, we adopt the clustering technique to group the
clients by their label distributions and collaboratively train a model inside
each cluster. We theoretically analyze the advantage of concatenation over
averaging by analyzing the information bottleneck of deep neural networks.
Experimental results demonstrate that FedConcat achieves significantly higher
accuracy than previous state-of-the-art FL methods in various heterogeneous
label skew distribution settings and meanwhile has lower communication costs.
Our code is publicly available at https://github.com/sjtudyq/FedConcat.
