  Numerous research studies in the field of federated learning (FL) have
attempted to use personalization to address the heterogeneity among clients,
one of FL's most crucial and challenging problems. However, existing works
predominantly focus on tailoring models. Yet, due to the heterogeneity of
clients, they may each require different choices of hyperparameters, which have
not been studied so far. We pinpoint two challenges of personalized federated
hyperparameter optimization (pFedHPO): handling the exponentially increased
search space and characterizing each client without compromising its data
privacy. To overcome them, we propose learning a
\textsc{H}yper\textsc{P}arameter \textsc{N}etwork (HPN) fed with client
encoding to decide personalized hyperparameters. The client encoding is
calculated with a random projection-based procedure to protect each client's
privacy. Besides, we design a novel mechanism to debias the low-fidelity
function evaluation samples for learning HPN. We conduct extensive experiments
on FL tasks from various domains, demonstrating the superiority of HPN.
