  The linearized-Laplace approximation (LLA) has been shown to be effective and
efficient in constructing Bayesian neural networks. It is theoretically
compelling since it can be seen as a Gaussian process posterior with the mean
function given by the neural network's maximum-a-posteriori predictive function
and the covariance function induced by the empirical neural tangent kernel.
However, while its efficacy has been studied in large-scale tasks like image
classification, it has not been studied in sequential decision-making problems
like Bayesian optimization where Gaussian processes -- with simple mean
functions and kernels such as the radial basis function -- are the de-facto
surrogate models. In this work, we study the usefulness of the LLA in Bayesian
optimization and highlight its strong performance and flexibility. However, we
also present some pitfalls that might arise and a potential problem with the
LLA when the search space is unbounded.
