  Though successful, federated learning presents new challenges for machine
learning, especially when the issue of data heterogeneity, also known as
Non-IID data, arises. To cope with the statistical heterogeneity, previous
works incorporated a proximal term in local optimization or modified the model
aggregation scheme at the server side or advocated clustered federated learning
approaches where the central server groups agent population into clusters with
jointly trainable data distributions to take the advantage of a certain level
of personalization. While effective, they lack a deep elaboration on what kind
of data heterogeneity and how the data heterogeneity impacts the accuracy
performance of the participating clients. In contrast to many of the prior
federated learning approaches, we demonstrate not only the issue of data
heterogeneity in current setups is not necessarily a problem but also in fact
it can be beneficial for the FL participants. Our observations are intuitive:
(1) Dissimilar labels of clients (label skew) are not necessarily considered
data heterogeneity, and (2) the principal angle between the agents' data
subspaces spanned by their corresponding principal vectors of data is a better
estimate of the data heterogeneity. Our code is available at
https://github.com/MMorafah/FL-SC-NIID.
