  We present a novel stochastic variational Gaussian process ($\mathcal{GP}$)
inference method, based on a posterior over a learnable set of weighted pseudo
input-output points (coresets). Instead of a free-form variational family, the
proposed coreset-based, variational tempered family for $\mathcal{GP}$s (CVTGP)
is defined in terms of the $\mathcal{GP}$ prior and the data-likelihood; hence,
accommodating the modeling inductive biases. We derive CVTGP's lower bound for
the log-marginal likelihood via marginalization of the proposed posterior over
latent $\mathcal{GP}$ coreset variables, and show it is amenable to stochastic
optimization. CVTGP reduces the learnable parameter size to $\mathcal{O}(M)$,
enjoys numerical stability, and maintains $\mathcal{O}(M^3)$ time- and
$\mathcal{O}(M^2)$ space-complexity, by leveraging a coreset-based tempered
posterior that, in turn, provides sparse and explainable representations of the
data. Results on simulated and real-world regression problems with Gaussian
observation noise validate that CVTGP provides better evidence lower-bound
estimates and predictive root mean squared error than alternative stochastic
$\mathcal{GP}$ inference methods.
