  Federated Averaging (FedAvg) is known to experience convergence issues when
encountering significant clients system heterogeneity and data heterogeneity.
Server momentum has been proposed as an effective mitigation. However, existing
server momentum works are restrictive in the momentum formulation, do not
properly schedule hyperparameters and focus only on system homogeneous
settings, which leaves the role of server momentum still an under-explored
problem. In this paper, we propose a general framework for server momentum,
that (a) covers a large class of momentum schemes that are unexplored in
federated learning (FL), (b) enables a popular stagewise hyperparameter
scheduler, (c) allows heterogeneous and asynchronous local computing. We
provide rigorous convergence analysis for the proposed framework. To our best
knowledge, this is the first work that thoroughly analyzes the performances of
server momentum with a hyperparameter scheduler and system heterogeneity.
Extensive experiments validate the effectiveness of our proposed framework.
