  The theoretical landscape of federated learning (FL) undergoes rapid
evolution, but its practical application encounters a series of intricate
challenges, and hyperparameter optimization is one of these critical
challenges. Amongst the diverse adjustments in hyperparameters, the adaptation
of the learning rate emerges as a crucial component, holding the promise of
significantly enhancing the efficacy of FL systems. In response to this
critical need, this paper presents FedHyper, a novel hypergradient-based
learning rate adaptation algorithm specifically designed for FL. FedHyper
serves as a universal learning rate scheduler that can adapt both global and
local rates as the training progresses. In addition, FedHyper not only
showcases unparalleled robustness to a spectrum of initial learning rate
configurations but also significantly alleviates the necessity for laborious
empirical learning rate adjustments. We provide a comprehensive theoretical
analysis of FedHyper's convergence rate and conduct extensive experiments on
vision and language benchmark datasets. The results demonstrate that FEDHYPER
consistently converges 1.1-3x faster than FedAvg and the competing baselines
while achieving superior final accuracy. Moreover, FedHyper catalyzes a
remarkable surge in accuracy, augmenting it by up to 15% compared to FedAvg
under suboptimal initial learning rate settings.
