  Reinforcement learning algorithms commonly seek to optimize policies for
solving one particular task. How should we explore an unknown dynamical system
such that the estimated model globally approximates the dynamics and allows us
to solve multiple downstream tasks in a zero-shot manner? In this paper, we
address this challenge, by developing an algorithm -- OPAX -- for active
exploration. OPAX uses well-calibrated probabilistic models to quantify the
epistemic uncertainty about the unknown dynamics. It optimistically -- w.r.t.
to plausible dynamics -- maximizes the information gain between the unknown
dynamics and state observations. We show how the resulting optimization problem
can be reduced to an optimal control problem that can be solved at each episode
using standard approaches. We analyze our algorithm for general models, and, in
the case of Gaussian process dynamics, we give a first-of-its-kind sample
complexity bound and show that the epistemic uncertainty converges to zero. In
our experiments, we compare OPAX with other heuristic active exploration
approaches on several environments. Our experiments show that OPAX is not only
theoretically sound but also performs well for zero-shot planning on novel
downstream tasks.
