  Federated Learning (FL) aims to train high-quality models in collaboration
with distributed clients while not uploading their local data, which attracts
increasing attention in both academia and industry. However, there is still a
considerable gap between the flourishing FL research and real-world scenarios,
mainly caused by the characteristics of heterogeneous devices and its scales.
Most existing works conduct evaluations with homogeneous devices, which are
mismatched with the diversity and variability of heterogeneous devices in
real-world scenarios. Moreover, it is challenging to conduct research and
development at scale with heterogeneous devices due to limited resources and
complex software stacks. These two key factors are important yet underexplored
in FL research as they directly impact the FL training dynamics and final
performance, making the effectiveness and usability of FL algorithms unclear.
To bridge the gap, in this paper, we propose an efficient and scalable
prototyping system for real-world cross-device FL, FS-Real. It supports
heterogeneous device runtime, contains parallelism and robustness enhanced FL
server, and provides implementations and extensibility for advanced FL utility
features such as personalization, communication compression and asynchronous
aggregation. To demonstrate the usability and efficiency of FS-Real, we conduct
extensive experiments with various device distributions, quantify and analyze
the effect of the heterogeneous device and various scales, and further provide
insights and open discussions about real-world FL scenarios. Our system is
released to help to pave the way for further real-world FL research and broad
applications involving diverse devices and scales.
