  The use of Gaussian process models is typically limited to datasets with a
few tens of thousands of observations due to their complexity and memory
footprint. The two most commonly used methods to overcome this limitation are
1) the variational sparse approximation which relies on inducing points and 2)
the state-space equivalent formulation of Gaussian processes which can be seen
as exploiting some sparsity in the precision matrix. We propose to take the
best of both worlds: we show that the inducing point framework is still valid
for state space models and that it can bring further computational and memory
savings. Furthermore, we provide the natural gradient formulation for the
proposed variational parameterisation. Finally, this work makes it possible to
use the state-space formulation inside deep Gaussian process models as
illustrated in one of the experiments.
