  Federated learning (FL) is an emerging, privacy-preserving machine learning
paradigm, drawing tremendous attention in both academia and industry. A unique
characteristic of FL is heterogeneity, which resides in the various hardware
specifications and dynamic states across the participating devices.
Theoretically, heterogeneity can exert a huge influence on the FL training
process, e.g., causing a device unavailable for training or unable to upload
its model updates. Unfortunately, these impacts have never been systematically
studied and quantified in existing FL literature.
  In this paper, we carry out the first empirical study to characterize the
impacts of heterogeneity in FL. We collect large-scale data from 136k
smartphones that can faithfully reflect heterogeneity in real-world settings.
We also build a heterogeneity-aware FL platform that complies with the standard
FL protocol but with heterogeneity in consideration. Based on the data and the
platform, we conduct extensive experiments to compare the performance of
state-of-the-art FL algorithms under heterogeneity-aware and
heterogeneity-unaware settings. Results show that heterogeneity causes
non-trivial performance degradation in FL, including up to 9.2% accuracy drop,
2.32x lengthened training time, and undermined fairness. Furthermore, we
analyze potential impact factors and find that device failure and participant
bias are two potential factors for performance degradation. Our study provides
insightful implications for FL practitioners. On the one hand, our findings
suggest that FL algorithm designers consider necessary heterogeneity during the
evaluation. On the other hand, our findings urge system providers to design
specific mechanisms to mitigate the impacts of heterogeneity.
