  Continuously learning to solve unseen tasks with limited experience has been
extensively pursued in meta-learning and continual learning, but with
restricted assumptions such as accessible task distributions, independently and
identically distributed tasks, and clear task delineations. However, real-world
physical tasks frequently violate these assumptions, resulting in performance
degradation. This paper proposes a continual online model-based reinforcement
learning approach that does not require pre-training to solve task-agnostic
problems with unknown task boundaries. We maintain a mixture of experts to
handle nonstationarity, and represent each different type of dynamics with a
Gaussian Process to efficiently leverage collected data and expressively model
uncertainty. We propose a transition prior to account for the temporal
dependencies in streaming data and update the mixture online via sequential
variational inference. Our approach reliably handles the task distribution
shift by generating new models for never-before-seen dynamics and reusing old
models for previously seen dynamics. In experiments, our approach outperforms
alternative methods in non-stationary tasks, including classic control with
changing dynamics and decision making in different driving scenarios.
