  Prior beliefs about the latent function to shape inductive biases can be
incorporated into a Gaussian Process (GP) via the kernel. However, beyond
kernel choices, the decision-making process of GP models remains poorly
understood. In this work, we contribute an analysis of the loss landscape for
GP models using methods from physics. We demonstrate $\nu$-continuity for
Matern kernels and outline aspects of catastrophe theory at critical points in
the loss landscape. By directly including $\nu$ in the hyperparameter
optimisation for Matern kernels, we find that typical values of $\nu$ are far
from optimal in terms of performance, yet prevail in the literature due to the
increased computational speed. We also provide an a priori method for
evaluating the effect of GP ensembles and discuss various voting approaches
based on physical properties of the loss landscape. The utility of these
approaches is demonstrated for various synthetic and real datasets. Our
findings provide an enhanced understanding of the decision-making process
behind GPs and offer practical guidance for improving their performance and
interpretability in a range of applications.
