  Personalized Federated Learning (PFL) is a commonly used framework that
allows clients to collaboratively train their personalized models. PFL is
particularly useful for handling situations where data from different clients
are not independent and identically distributed (non-IID). Previous research in
PFL implicitly assumes that clients can gain more benefits from those with
similar data distributions. Correspondingly, methods such as personalized
weight aggregation are developed to assign higher weights to similar clients
during training. We pose a question: can a client benefit from other clients
with dissimilar data distributions and if so, how? This question is
particularly relevant in scenarios with a high degree of non-IID, where clients
have widely different data distributions, and learning from only similar
clients will lose knowledge from many other clients. We note that when dealing
with clients with similar data distributions, methods such as personalized
weight aggregation tend to enforce their models to be close in the parameter
space. It is reasonable to conjecture that a client can benefit from dissimilar
clients if we allow their models to depart from each other. Based on this idea,
we propose DiversiFed which allows each client to learn from clients with
diversified data distribution in personalized federated learning. DiversiFed
pushes personalized models of clients with dissimilar data distributions apart
in the parameter space while pulling together those with similar distributions.
In addition, to achieve the above effect without using prior knowledge of data
distribution, we design a loss function that leverages the model similarity to
determine the degree of attraction and repulsion between any two models.
Experiments on several datasets show that DiversiFed can benefit from
dissimilar clients and thus outperform the state-of-the-art methods.
