  Bayesian optimization is an elegant solution to the hyperparameter
optimization problem in machine learning. Building a reliable and robust
Bayesian optimization service requires careful testing methodology and sound
statistical analysis. In this talk we will outline our development of an
evaluation framework to rigorously test and measure the impact of changes to
the SigOpt optimization service. We present an overview of our evaluation
system and discuss how this framework empowers our research engineers to
confidently and quickly make changes to our core optimization engine
