  Federated learning (FL) has emerged as a widely adopted training paradigm for
privacy-preserving machine learning. While the SGD-based FL algorithms have
demonstrated considerable success in the past, there is a growing trend towards
adopting adaptive federated optimization methods, particularly for training
large-scale models. However, the conventional synchronous aggregation design
poses a significant challenge to the practical deployment of those adaptive
federated optimization methods, particularly in the presence of straggler
clients. To fill this research gap, this paper introduces federated adaptive
asynchronous optimization, named FADAS, a novel method that incorporates
asynchronous updates into adaptive federated optimization with provable
guarantees. To further enhance the efficiency and resilience of our proposed
method in scenarios with significant asynchronous delays, we also extend FADAS
with a delay-adaptive learning adjustment strategy. We rigorously establish the
convergence rate of the proposed algorithms and empirical results demonstrate
the superior performance of FADAS over other asynchronous FL baselines.
