  Hyper-parameters optimization (HPO) is vital for machine learning models.
Besides model accuracy, other tuning intentions such as model training time and
energy consumption are also worthy of attention from data analytic service
providers. Hence, it is essential to take both model hyperparameters and system
parameters into consideration to execute cross-layer multi-objective
hyperparameter auto-tuning. Towards this challenging target, we propose
HyperTuner in this paper. To address the formulated high-dimensional black-box
multi-objective optimization problem, HyperTuner first conducts multi-objective
parameter importance ranking with its MOPIR algorithm and then leverages the
proposed ADUMBO algorithm to find the Pareto-optimal configuration set. During
each iteration, ADUMBO selects the most promising configuration from the
generated Pareto candidate set via maximizing a new well-designed metric, which
can adaptively leverage the uncertainty as well as the predicted mean across
all the surrogate models along with the iteration times. We evaluate HyperTuner
on our local distributed TensorFlow cluster and experimental results show that
it is always able to find a better Pareto configuration front superior in both
convergence and diversity compared with the other four baseline algorithms.
Besides, experiments with different training datasets, different optimization
objectives and different machine learning platforms verify that HyperTuner can
well adapt to various data analytic service scenarios.
