  Model-based reinforcement learning algorithms with probabilistic dynamical
models are amongst the most data-efficient learning methods. This is often
attributed to their ability to distinguish between epistemic and aleatoric
uncertainty. However, while most algorithms distinguish these two uncertainties
for learning the model, they ignore it when optimizing the policy, which leads
to greedy and insufficient exploration. At the same time, there are no
practical solvers for optimistic exploration algorithms. In this paper, we
propose a practical optimistic exploration algorithm (H-UCRL). H-UCRL
reparameterizes the set of plausible models and hallucinates control directly
on the epistemic uncertainty. By augmenting the input space with the
hallucinated inputs, H-UCRL can be solved using standard greedy planners.
Furthermore, we analyze H-UCRL and construct a general regret bound for
well-calibrated models, which is provably sublinear in the case of Gaussian
Process models. Based on this theoretical foundation, we show how optimistic
exploration can be easily combined with state-of-the-art reinforcement learning
algorithms and different probabilistic models. Our experiments demonstrate that
optimistic exploration significantly speeds-up learning when there are
penalties on actions, a setting that is notoriously difficult for existing
model-based reinforcement learning algorithms.
