  Federated Learning (FL) is a distributed learning scheme to train a shared
model across clients. One common and fundamental challenge in FL is that the
sets of data across clients could be non-identically distributed and have
different sizes. Personalized Federated Learning (PFL) attempts to solve this
challenge via locally adapted models. In this work, we present a novel
framework for PFL based on hierarchical Bayesian modeling and variational
inference. A global model is introduced as a latent variable to augment the
joint distribution of clients' parameters and capture the common trends of
different clients, optimization is derived based on the principle of maximizing
the marginal likelihood and conducted using variational expectation
maximization. Our algorithm gives rise to a closed-form estimation of a
confidence value which comprises the uncertainty of clients' parameters and
local model deviations from the global model. The confidence value is used to
weigh clients' parameters in the aggregation stage and adjust the
regularization effect of the global model. We evaluate our method through
extensive empirical studies on multiple datasets. Experimental results show
that our approach obtains competitive results under mild heterogeneous
circumstances while significantly outperforming state-of-the-art PFL frameworks
in highly heterogeneous settings. Our code is available at
https://github.com/JunyiZhu-AI/confidence_aware_PFL.
