  The recent trend towards Personalized Federated Learning (PFL) has garnered
significant attention as it allows for the training of models that are tailored
to each client while maintaining data privacy. However, current PFL techniques
primarily focus on modeling the conditional distribution heterogeneity (i.e.
concept shift), which can result in suboptimal performance when the
distribution of input data across clients diverges (i.e. covariate shift).
Additionally, these techniques often lack the ability to adapt to unseen data,
further limiting their effectiveness in real-world scenarios. To address these
limitations, we propose a novel approach, FedGMM, which utilizes Gaussian
mixture models (GMM) to effectively fit the input data distributions across
diverse clients. The model parameters are estimated by maximum likelihood
estimation utilizing a federated Expectation-Maximization algorithm, which is
solved in closed form and does not assume gradient similarity. Furthermore,
FedGMM possesses an additional advantage of adapting to new clients with
minimal overhead, and it also enables uncertainty quantification. Empirical
evaluations on synthetic and benchmark datasets demonstrate the superior
performance of our method in both PFL classification and novel sample
detection.
