  Hyperparameter optimization (HPO) is crucial for machine learning algorithms
to achieve satisfactory performance, whose progress has been boosted by related
benchmarks. Nonetheless, existing efforts in benchmarking all focus on HPO for
traditional centralized learning while ignoring federated learning (FL), a
promising paradigm for collaboratively learning models from dispersed data. In
this paper, we first identify some uniqueness of HPO for FL algorithms from
various aspects. Due to this uniqueness, existing HPO benchmarks no longer
satisfy the need to compare HPO methods in the FL setting. To facilitate the
research of HPO in the FL setting, we propose and implement a benchmark suite
FedHPO-B that incorporates comprehensive FL tasks, enables efficient function
evaluations, and eases continuing extensions. We also conduct extensive
experiments based on FedHPO-B to benchmark a few HPO methods. We open-source
FedHPO-B at
https://github.com/alibaba/FederatedScope/tree/master/benchmark/FedHPOB.
