  Federated Learning (FL) is a distributed machine learning (ML) paradigm, in
which multiple clients collaboratively train ML models without centralizing
their local data. Similar to conventional ML pipelines, the client local
optimization and server aggregation procedure in FL are sensitive to the
hyperparameter (HP) selection. Despite extensive research on tuning HPs for
centralized ML, these methods yield suboptimal results when employed in FL.
This is mainly because their "training-after-tuning" framework is unsuitable
for FL with limited client computation power. While some approaches have been
proposed for HP-Tuning in FL, they are limited to the HPs for client local
updates. In this work, we propose a novel HP-tuning algorithm, called Federated
Population-based Hyperparameter Tuning (FedPop), to address this vital yet
challenging problem. FedPop employs population-based evolutionary algorithms to
optimize the HPs, which accommodates various HP types at both the client and
server sides. Compared with prior tuning methods, FedPop employs an online
"tuning-while-training" framework, offering computational efficiency and
enabling the exploration of a broader HP search space. Our empirical validation
on the common FL benchmarks and complex real-world FL datasets, including
full-sized Non-IID ImageNet-1K, demonstrates the effectiveness of the proposed
method, which substantially outperforms the concurrent state-of-the-art
HP-tuning methods in FL.
