  Model averaging is a widely adopted technique in federated learning (FL) that
aggregates multiple client models to obtain a global model. Remarkably, model
averaging in FL yields a superior global model, even when client models are
trained with non-convex objective functions and on heterogeneous local
datasets. However, the rationale behind its success remains poorly understood.
To shed light on this issue, we first visualize the loss landscape of FL over
client and global models to illustrate their geometric properties. The
visualization shows that the client models encompass the global model within a
common basin, and interestingly, the global model may deviate from the basin's
center while still outperforming the client models. To gain further insights
into model averaging in FL, we decompose the expected loss of the global model
into five factors related to the client models. Specifically, our analysis
reveals that the global model loss after early training mainly arises from
\textit{i)} the client model's loss on non-overlapping data between client
datasets and the global dataset and \textit{ii)} the maximum distance between
the global and client models. Based on the findings from our loss landscape
visualization and loss decomposition, we propose utilizing iterative moving
averaging (IMA) on the global model at the late training phase to reduce its
deviation from the expected minimum, while constraining client exploration to
limit the maximum distance between the global and client models. Our
experiments demonstrate that incorporating IMA into existing FL methods
significantly improves their accuracy and training speed on various
heterogeneous data setups of benchmark datasets. Code is available at
\url{https://github.com/TailinZhou/FedIMA}.
