  Practical Bayesian learning often requires (1) online inference, (2) dynamic
models, and (3) ensembling over multiple different models. Recent advances have
shown how to use random feature approximations to achieve scalable, online
ensembling of Gaussian processes with desirable theoretical properties and
fruitful applications. One key to these methods' success is the inclusion of a
random walk on the model parameters, which makes models dynamic. We show that
these methods can be generalized easily to any basis expansion model and that
using alternative basis expansions, such as Hilbert space Gaussian processes,
often results in better performance. To simplify the process of choosing a
specific basis expansion, our method's generality also allows the ensembling of
several entirely different models, for example, a Gaussian process and
polynomial regression. Finally, we propose a novel method to ensemble static
and dynamic models together.
