  Many state-of-the-art hyperparameter optimization (HPO) algorithms rely on
model-based optimizers that learn surrogate models of the target function to
guide the search. Gaussian processes are the de facto surrogate model due to
their ability to capture uncertainty but they make strong assumptions about the
observation noise, which might not be warranted in practice. In this work, we
propose to leverage conformalized quantile regression which makes minimal
assumptions about the observation noise and, as a result, models the target
function in a more realistic and robust fashion which translates to quicker HPO
convergence on empirical benchmarks. To apply our method in a multi-fidelity
setting, we propose a simple, yet effective, technique that aggregates observed
results across different resource levels and outperforms conventional methods
across many empirical tasks.
