  The reliance of language model training on massive amounts of computation and
vast datasets scraped from potentially low-quality, copyrighted, or sensitive
data has come into question practically, legally, and ethically. Federated
learning provides a plausible alternative by enabling previously untapped data
to be voluntarily gathered from collaborating organizations. However, when
scaled globally, federated learning requires collaboration across heterogeneous
legal, security, and privacy regimes while accounting for the inherent locality
of language data; this further exacerbates the established challenge of
federated statistical heterogeneity. We propose a Worldwide Federated Language
Model Training~(WorldLM) system based on federations of federations, where each
federation has the autonomy to account for factors such as its industry,
operating jurisdiction, or competitive environment. WorldLM enables such
autonomy in the presence of statistical heterogeneity via partial model
localization by allowing sub-federations to attentively aggregate key layers
from their constituents. Furthermore, it can adaptively share information
across federations via residual layer embeddings. Evaluations of language
modeling on naturally heterogeneous datasets show that WorldLM outperforms
standard federations by up to $1.91\times$, approaches the personalized
performance of fully local models, and maintains these advantages under
privacy-enhancing techniques.
