  Model selection is an integral problem of model based optimization techniques
such as Bayesian optimization (BO). Current approaches often treat model
selection as an estimation problem, to be periodically updated with
observations coming from the optimization iterations. In this paper, we propose
an alternative way to achieve both efficiently. Specifically, we propose a
novel way of integrating model selection and BO for the single goal of reaching
the function optima faster. The algorithm moves back and forth between BO in
the model space and BO in the function space, where the goodness of the
recommended model is captured by a score function and fed back, capturing how
well the model helped convergence in the function space. The score function is
derived in such a way that it neutralizes the effect of the moving nature of
the BO in the function space, thus keeping the model selection problem
stationary. This back and forth leads to quick convergence for both model
selection and BO in the function space. In addition to improved sample
efficiency, the framework outputs information about the black-box function.
Convergence is proved, and experimental results show significant improvement
compared to standard BO.
