  We consider the problem of personalized federated learning when there are
known cluster structures within users. An intuitive approach would be to
regularize the parameters so that users in the same cluster share similar model
weights. The distances between the clusters can then be regularized to reflect
the similarity between different clusters of users. We develop an algorithm
that allows each cluster to communicate independently and derive the
convergence results. We study a hierarchical linear model to theoretically
demonstrate that our approach outperforms agents learning independently and
agents learning a single shared weight. Finally, we demonstrate the advantages
of our approach using both simulated and real-world data.
