  Federated learning has attracted increasing attention with the emergence of
distributed data. While extensive federated learning algorithms have been
proposed for the non-convex distributed problem, federated learning in practice
still faces numerous challenges, such as the large training iterations to
converge since the sizes of models and datasets keep increasing, and the lack
of adaptivity by SGD-based model updates. Meanwhile, the study of adaptive
methods in federated learning is scarce and existing works either lack a
complete theoretical convergence guarantee or have slow sample complexity. In
this paper, we propose an efficient adaptive algorithm (i.e., FAFED) based on
the momentum-based variance-reduced technique in cross-silo FL. We first
explore how to design the adaptive algorithm in the FL setting. By providing a
counter-example, we prove that a simple combination of FL and adaptive methods
could lead to divergence. More importantly, we provide a convergence analysis
for our method and prove that our algorithm is the first adaptive FL algorithm
to reach the best-known samples $O(\epsilon^{-3})$ and $O(\epsilon^{-2})$
communication rounds to find an $\epsilon$-stationary point without large
batches. The experimental results on the language modeling task and image
classification task with heterogeneous data demonstrate the efficiency of our
algorithms.
