  Personalized federated learning considers learning models unique to each
client in a heterogeneous network. The resulting client-specific models have
been purported to improve metrics such as accuracy, fairness, and robustness in
federated networks. However, despite a plethora of work in this area, it
remains unclear: (1) which personalization techniques are most effective in
various settings, and (2) how important personalization truly is for realistic
federated applications. To better answer these questions, we propose Motley, a
benchmark for personalized federated learning. Motley consists of a suite of
cross-device and cross-silo federated datasets from varied problem domains, as
well as thorough evaluation metrics for better understanding the possible
impacts of personalization. We establish baselines on the benchmark by
comparing a number of representative personalized federated learning methods.
These initial results highlight strengths and weaknesses of existing
approaches, and raise several open questions for the community. Motley aims to
provide a reproducible means with which to advance developments in personalized
and heterogeneity-aware federated learning, as well as the related areas of
transfer learning, meta-learning, and multi-task learning.
