  Gaussian processes (GPs) are the most common formalism for defining
probability distributions over spaces of functions. While applications of GPs
are myriad, a comprehensive understanding of GP sample paths, i.e. the function
spaces over which they define a probability measure, is lacking. In practice,
GPs are not constructed through a probability measure, but instead through a
mean function and a covariance kernel. In this paper we provide necessary and
sufficient conditions on the covariance kernel for the sample paths of the
corresponding GP to attain a given regularity. We use the framework of H\"older
regularity as it grants particularly straightforward conditions, which simplify
further in the cases of stationary and isotropic GPs. We then demonstrate that
our results allow for novel and unusually tight characterisations of the sample
path regularities of the GPs commonly used in machine learning applications,
such as the Mat\'ern GPs.
