  The design of machine learning systems often requires trading off different
objectives, for example, prediction error and energy consumption for deep
neural networks (DNNs). Typically, no single design performs well in all
objectives; therefore, finding Pareto-optimal designs is of interest. The
search for Pareto-optimal designs involves evaluating designs in an iterative
process, and the measurements are used to evaluate an acquisition function that
guides the search process. However, measuring different objectives incurs
different costs. For example, the cost of measuring the prediction error of
DNNs is orders of magnitude higher than that of measuring the energy
consumption of a pre-trained DNN, as it requires re-training the DNN. Current
state-of-the-art methods do not consider this difference in objective
evaluation cost, potentially incurring expensive evaluations of objective
functions in the optimization process. In this paper, we develop a novel
decoupled and cost-aware multi-objective optimization algorithm, we call
Flexible Multi-Objective Bayesian Optimization (FlexiBO) to address this issue.
FlexiBO weights the improvement of the hypervolume of the Pareto region by the
measurement cost of each objective to balance the expense of collecting new
information with the knowledge gained through objective evaluations, preventing
us from performing expensive measurements for little to no gain. We evaluate
FlexiBO on seven state-of-the-art DNNs for image recognition, natural language
processing (NLP), and speech-to-text translation. Our results indicate that,
given the same total experimental budget, FlexiBO discovers designs with
4.8$\%$ to 12.4$\%$ lower hypervolume error than the best method in
state-of-the-art multi-objective optimization.
