  Gaussian processes (GPs) are an important tool in machine learning and
statistics with applications ranging from social and natural science through
engineering. They constitute a powerful kernelized non-parametric method with
well-calibrated uncertainty estimates, however, off-the-shelf GP inference
procedures are limited to datasets with several thousand data points because of
their cubic computational complexity. For this reason, many sparse GPs
techniques have been developed over the past years. In this paper, we focus on
GP regression tasks and propose a new approach based on aggregating predictions
from several local and correlated experts. Thereby, the degree of correlation
between the experts can vary between independent up to fully correlated
experts. The individual predictions of the experts are aggregated taking into
account their correlation resulting in consistent uncertainty estimates. Our
method recovers independent Product of Experts, sparse GP and full GP in the
limiting cases. The presented framework can deal with a general kernel function
and multiple variables, and has a time and space complexity which is linear in
the number of experts and data samples, which makes our approach highly
scalable. We demonstrate superior performance, in a time vs. accuracy sense, of
our proposed method against state-of-the-art GP approximation methods for
synthetic as well as several real-world datasets with deterministic and
stochastic optimization.
