  Gaussian processes (GPs) are powerful non-parametric function estimators.
However, their applications are largely limited by the expensive computational
cost of the inference procedures. Existing stochastic or distributed
synchronous variational inferences, although have alleviated this issue by
scaling up GPs to millions of samples, are still far from satisfactory for
real-world large applications, where the data sizes are often orders of
magnitudes larger, say, billions. To solve this problem, we propose ADVGP, the
first Asynchronous Distributed Variational Gaussian Process inference for
regression, on the recent large-scale machine learning platform,
PARAMETERSERVER. ADVGP uses a novel, flexible variational framework based on a
weight space augmentation, and implements the highly efficient, asynchronous
proximal gradient optimization. While maintaining comparable or better
predictive performance, ADVGP greatly improves upon the efficiency of the
existing variational methods. With ADVGP, we effortlessly scale up GP
regression to a real-world application with billions of samples and demonstrate
an excellent, superior prediction accuracy to the popular linear models.
