  Federated learning (FL) is a popular technique to train machine learning (ML)
models with decentralized data. Extensive works have studied the performance of
the global model; however, it is still unclear how the training process affects
the final test accuracy. Exacerbating this problem is the fact that FL
executions differ significantly from traditional ML with heterogeneous data
characteristics across clients, involving more hyperparameters. In this work,
we show that the final test accuracy of FL is dramatically affected by the
early phase of the training process, i.e., FL exhibits critical learning
periods, in which small gradient errors can have irrecoverable impact on the
final test accuracy. To further explain this phenomenon, we generalize the
trace of the Fisher Information Matrix (FIM) to FL and define a new notion
called FedFIM, a quantity reflecting the local curvature of each clients from
the beginning of the training in FL. Our findings suggest that the {\em initial
learning phase} plays a critical role in understanding the FL performance. This
is in contrast to many existing works which generally do not connect the final
accuracy of FL to the early phase training. Finally, seizing critical learning
periods in FL is of independent interest and could be useful for other problems
such as the choices of hyperparameters such as the number of client selected
per round, batch size, and more, so as to improve the performance of FL
training and testing.
