  This article revisits the fundamental problem of parameter selection for
Gaussian process interpolation. By choosing the mean and the covariance
functions of a Gaussian process within parametric families, the user obtains a
family of Bayesian procedures to perform predictions about the unknown
function, and must choose a member of the family that will hopefully provide
good predictive performances. We base our study on the general concept of
scoring rules, which provides an effective framework for building leave-one-out
selection and validation criteria, and a notion of extended likelihood criteria
based on an idea proposed by Fasshauer and co-authors in 2009, which makes it
possible to recover standard selection criteria such as, for instance, the
generalized cross-validation criterion. Under this setting, we empirically show
on several test problems of the literature that the choice of an appropriate
family of models is often more important than the choice of a particular
selection criterion (e.g., the likelihood versus a leave-one-out selection
criterion). Moreover, our numerical results show that the regularity parameter
of a Mat{\'e}rn covariance can be selected effectively by most selection
criteria.
