  Gaussian Processes are widely used for regression tasks. A known limitation
in the application of Gaussian Processes to regression tasks is that the
computation of the solution requires performing a matrix inversion. The
solution also requires the storage of a large matrix in memory. These factors
restrict the application of Gaussian Process regression to small and moderate
size data sets. We present an algorithm that combines estimates from models
developed using subsets of the data obtained in a manner similar to the
bootstrap. The sample size is a critical parameter for this algorithm.
Guidelines for reasonable choices of algorithm parameters, based on detailed
experimental study, are provided. Various techniques have been proposed to
scale Gaussian Processes to large scale regression tasks. The most appropriate
choice depends on the problem context. The proposed method is most appropriate
for problems where an additive model works well and the response depends on a
small number of features. The minimax rate of convergence for such problems is
attractive and we can build effective models with a small subset of the data.
The Stochastic Variational Gaussian Process and the Sparse Gaussian Process are
also appropriate choices for such problems. These methods pick a subset of data
based on theoretical considerations. The proposed algorithm uses bagging and
random sampling. Results from experiments conducted as part of this study
indicate that the algorithm presented in this work can be as effective as these
methods. Model stacking can be used to combine the model developed with the
proposed method with models from other methods for large scale regression such
as Gradient Boosted Trees. This can yield performance gains.
