  The performance of many machine learning models depends on their
hyper-parameter settings. Bayesian Optimization has become a successful tool
for hyper-parameter optimization of machine learning algorithms, which aims to
identify optimal hyper-parameters during an iterative sequential process.
However, most of the Bayesian Optimization algorithms are designed to select
models for effectiveness only and ignore the important issue of model training
efficiency. Given that both model effectiveness and training time are important
for real-world applications, models selected for effectiveness may not meet the
strict training time requirements necessary to deploy in a production
environment. In this work, we present a unified Bayesian Optimization framework
for jointly optimizing models for both prediction effectiveness and training
efficiency. We propose an objective that captures the tradeoff between these
two metrics and demonstrate how we can jointly optimize them in a principled
Bayesian Optimization framework. Experiments on model selection for
recommendation tasks indicate models selected this way significantly improves
model training efficiency while maintaining strong effectiveness as compared to
state-of-the-art Bayesian Optimization algorithms.
