  Bayesian optimization is a highly efficient approach to optimizing objective
functions which are expensive to query. These objectives are typically
represented by Gaussian process (GP) surrogate models which are easy to
optimize and support exact inference. While standard GP surrogates have been
well-established in Bayesian optimization, Bayesian neural networks (BNNs) have
recently become practical function approximators, with many benefits over
standard GPs such as the ability to naturally handle non-stationarity and learn
representations for high-dimensional data. In this paper, we study BNNs as
alternatives to standard GP surrogates for optimization. We consider a variety
of approximate inference procedures for finite-width BNNs, including
high-quality Hamiltonian Monte Carlo, low-cost stochastic MCMC, and heuristics
such as deep ensembles. We also consider infinite-width BNNs, linearized
Laplace approximations, and partially stochastic models such as deep kernel
learning. We evaluate this collection of surrogate models on diverse problems
with varying dimensionality, number of objectives, non-stationarity, and
discrete and continuous inputs. We find: (i) the ranking of methods is highly
problem dependent, suggesting the need for tailored inductive biases; (ii) HMC
is the most successful approximate inference procedure for fully stochastic
BNNs; (iii) full stochasticity may be unnecessary as deep kernel learning is
relatively competitive; (iv) deep ensembles perform relatively poorly; (v)
infinite-width BNNs are particularly promising, especially in high dimensions.
