  Conventional federated learning algorithms train a single global model by
leveraging all participating clients' data. However, due to heterogeneity in
client generative distributions and predictive models, these approaches may not
appropriately approximate the predictive process, converge to an optimal state,
or generalize to new clients. We study personalization and generalization in
stateless cross-device federated learning setups assuming heterogeneity in
client data distributions and predictive models. We first propose a
hierarchical generative model and formalize it using Bayesian Inference. We
then approximate this process using Variational Inference to train our model
efficiently. We call this algorithm Federated Variational Inference (FedVI). We
use PAC-Bayes analysis to provide generalization bounds for FedVI. We evaluate
our model on FEMNIST and CIFAR-100 image classification and show that FedVI
beats the state-of-the-art on both tasks.
