  Federated learning data is drawn from a distribution of distributions:
clients are drawn from a meta-distribution, and their data are drawn from local
data distributions. Thus generalization studies in federated learning should
separate performance gaps from unseen client data (out-of-sample gap) from
performance gaps from unseen client distributions (participation gap). In this
work, we propose a framework for disentangling these performance gaps. Using
this framework, we observe and explain differences in behavior across natural
and synthetic federated datasets, indicating that dataset synthesis strategy
can be important for realistic simulations of generalization in federated
learning. We propose a semantic synthesis strategy that enables realistic
simulation without naturally-partitioned data. Informed by our findings, we
call out community suggestions for future federated learning works.
