  Gaussian Process Regression (GPR) is widely used in statistics and machine
learning for prediction tasks requiring uncertainty measures. Its efficacy
depends on the appropriate specification of the mean function, covariance
kernel function, and associated hyperparameters. Severe misspecifications can
lead to inaccurate results and problematic consequences, especially in
safety-critical applications. However, a systematic approach to handle these
misspecifications is lacking in the literature. In this work, we propose a
general framework to address these issues. Firstly, we introduce a flexible
two-stage GPR framework that separates mean prediction and uncertainty
quantification (UQ) to prevent mean misspecification, which can introduce bias
into the model. Secondly, kernel function misspecification is addressed through
a novel automatic kernel search algorithm, supported by theoretical analysis,
that selects the optimal kernel from a candidate set. Additionally, we propose
a subsampling-based warm-start strategy for hyperparameter initialization to
improve efficiency and avoid hyperparameter misspecification. With much lower
computational cost, our subsampling-based strategy can yield competitive or
better performance than training exclusively on the full dataset. Combining all
these components, we recommend two GPR methods-exact and scalable-designed to
match available computational resources and specific UQ requirements. Extensive
evaluation on real-world datasets, including UCI benchmarks and a
safety-critical medical case study, demonstrates the robustness and precision
of our methods.
