  Bayesian optimization (BO) is a popular methodology to tune the
hyperparameters of expensive black-box functions. Traditionally, BO focuses on
a single task at a time and is not designed to leverage information from
related functions, such as tuning performance objectives of the same algorithm
across multiple datasets. In this work, we introduce a novel approach to
achieve transfer learning across different \emph{datasets} as well as different
\emph{objectives}. The main idea is to regress the mapping from hyperparameter
to objective quantiles with a semi-parametric Gaussian Copula distribution,
which provides robustness against different scales or outliers that can occur
in different tasks. We introduce two methods to leverage this mapping: a
Thompson sampling strategy as well as a Gaussian Copula process using such
quantile estimate as a prior. We show that these strategies can combine the
estimation of multiple objectives such as latency and accuracy, steering the
hyperparameters optimization toward faster predictions for the same level of
accuracy. Extensive experiments demonstrate significant improvements over
state-of-the-art methods for both hyperparameter optimization and neural
architecture search.
