  Federated learning aims to train models collaboratively across different
clients without the sharing of data for privacy considerations. However, one
major challenge for this learning paradigm is the {\em data heterogeneity}
problem, which refers to the discrepancies between the local data distributions
among various clients. To tackle this problem, we first study how data
heterogeneity affects the representations of the globally aggregated models.
Interestingly, we find that heterogeneous data results in the global model
suffering from severe {\em dimensional collapse}, in which representations tend
to reside in a lower-dimensional space instead of the ambient space. Moreover,
we observe a similar phenomenon on models locally trained on each client and
deduce that the dimensional collapse on the global model is inherited from
local models. In addition, we theoretically analyze the gradient flow dynamics
to shed light on how data heterogeneity result in dimensional collapse for
local models. To remedy this problem caused by the data heterogeneity, we
propose {\sc FedDecorr}, a novel method that can effectively mitigate
dimensional collapse in federated learning. Specifically, {\sc FedDecorr}
applies a regularization term during local training that encourages different
dimensions of representations to be uncorrelated. {\sc FedDecorr}, which is
implementation-friendly and computationally-efficient, yields consistent
improvements over baselines on standard benchmark datasets. Code:
https://github.com/bytedance/FedDecorr.
