  Hyperparameter optimization (HPO) is increasingly used to automatically tune
the predictive performance (e.g., accuracy) of machine learning models.
However, in a plethora of real-world applications, accuracy is only one of the
multiple -- often conflicting -- performance criteria, necessitating the
adoption of a multi-objective (MO) perspective. While the literature on MO
optimization is rich, few prior studies have focused on HPO. In this paper, we
propose algorithms that extend asynchronous successive halving (ASHA) to the MO
setting. Considering multiple evaluation metrics, we assess the performance of
these methods on three real world tasks: (i) Neural architecture search, (ii)
algorithmic fairness and (iii) language model optimization. Our empirical
analysis shows that MO ASHA enables to perform MO HPO at scale. Further, we
observe that that taking the entire Pareto front into account for candidate
selection consistently outperforms multi-fidelity HPO based on MO scalarization
in terms of wall-clock time. Our algorithms (to be open-sourced) establish new
baselines for future research in the area.
