  Dyna-style model-based reinforcement learning (MBRL) combines model-free
agents with predictive transition models through model-based rollouts. This
combination raises a critical question: 'When to trust your model?'; i.e.,
which rollout length results in the model providing useful data? Janner et al.
(2019) address this question by gradually increasing rollout lengths throughout
the training. While theoretically tempting, uniform model accuracy is a fallacy
that collapses at the latest when extrapolating. Instead, we propose asking the
question 'Where to trust your model?'. Using inherent model uncertainty to
consider local accuracy, we obtain the Model-Based Actor-Critic with
Uncertainty-Aware Rollout Adaption (MACURA) algorithm. We propose an
easy-to-tune rollout mechanism and demonstrate substantial improvements in data
efficiency and performance compared to state-of-the-art deep MBRL methods on
the MuJoCo benchmark.
