  Federated learning (FL) is a promising approach that enables distributed
clients to collaboratively train a global model while preserving their data
privacy. However, FL often suffers from data heterogeneity problems, which can
significantly affect its performance. To address this, clustered federated
learning (CFL) has been proposed to construct personalized models for different
client clusters. One effective client clustering strategy is to allow clients
to choose their own local models from a model pool based on their performance.
However, without pre-trained model parameters, such a strategy is prone to
clustering failure, in which all clients choose the same model. Unfortunately,
collecting a large amount of labeled data for pre-training can be costly and
impractical in distributed environments. To overcome this challenge, we
leverage self-supervised contrastive learning to exploit unlabeled data for the
pre-training of FL systems. Together, self-supervised pre-training and client
clustering can be crucial components for tackling the data heterogeneity issues
of FL. Leveraging these two crucial strategies, we propose contrastive
pre-training-based clustered federated learning (CP-CFL) to improve the model
convergence and overall performance of FL systems. In this work, we demonstrate
the effectiveness of CP-CFL through extensive experiments in heterogeneous FL
settings, and present various interesting observations.
