  Federated learning (FL) is a distributed learning framework where numerous
clients collaborate with a central server to train a model without sharing
local data. However, the standard federated optimization in real-world
applications faces both statistical and system heterogeneity challenges, which
result in unfavorable convergence behavior. The previous works attempted to
modify the local training process (client-side) to tackle heterogeneity
challenges. However, they ignored that the updates on the server side can
coordinate the diverse local updates efficiently. This work explores the effect
of server-side updates against heterogeneity issues. We first introduce the
gradient diversity maximization direction findings, suggesting the global model
moves continuously in this direction for fast and stable convergence. Then, we
derive a novel server-side optimizer \textsc{FedAWARE} with rigorous
convergence analysis for general non-convex settings. Our extensive experiments
across multiple heterogeneous federated settings using four datasets showcase
that \textsc{FedAWARE} achieves competitive convergence performance in
comparison to state-of-the-art adaptive federated optimizers. Furthermore, our
results show that \textsc{FedAWARE} can enhance the performance of FL
algorithms as a plug-in module. Our source code is available at
\url{https://github.com/dunzeng/FedAWARE}.
