  Hyperparameter tuning is critical to the success of federated learning
applications. Unfortunately, appropriately selecting hyperparameters is
challenging in federated networks. Issues of scale, privacy, and heterogeneity
introduce noise in the tuning process and make it difficult to evaluate the
performance of various hyperparameters. In this work, we perform the first
systematic study on the effect of noisy evaluation in federated hyperparameter
tuning. We first identify and rigorously explore key sources of noise,
including client subsampling, data and systems heterogeneity, and data privacy.
Surprisingly, our results indicate that even small amounts of noise can
significantly impact tuning methods-reducing the performance of
state-of-the-art approaches to that of naive baselines. To address noisy
evaluation in such scenarios, we propose a simple and effective approach that
leverages public proxy data to boost the evaluation signal. Our work
establishes general challenges, baselines, and best practices for future work
in federated hyperparameter tuning.
