  Bayesian posterior distributions arising in modern applications, including
inverse problems in partial differential equation models in tomography and
subsurface flow, are often computationally intractable due to the large
computational cost of evaluating the data likelihood. To alleviate this
problem, we consider using Gaussian process regression to build a surrogate
model for the likelihood, resulting in an approximate posterior distribution
that is amenable to computations in practice. This work serves as an
introduction to Gaussian process regression, in particular in the context of
building surrogate models for inverse problems, and presents new insights into
a suitable choice of training points. We show that the error between the true
and approximate posterior distribution can be bounded by the error between the
true and approximate likelihood, measured in the $L^2$-norm weighted by the
true posterior, and that efficiently bounding the error between the true and
approximate likelihood in this norm suggests choosing the training points in
the Gaussian process surrogate model based on the true posterior.
