  Dyna is a fundamental approach to model-based reinforcement learning (MBRL)
that interleaves planning, acting, and learning in an online setting. In the
most typical application of Dyna, the dynamics model is used to generate
one-step transitions from selected start states from the agent's history, which
are used to update the agent's value function or policy as if they were real
experiences. In this work, one-step Dyna was applied to several games from the
Arcade Learning Environment (ALE). We found that the model-based updates
offered surprisingly little benefit over simply performing more updates with
the agent's existing experience, even when using a perfect model. We
hypothesize that to get the most from planning, the model must be used to
generate unfamiliar experience. To test this, we experimented with the "shape"
of planning in multiple different concrete instantiations of Dyna, performing
fewer, longer rollouts, rather than many short rollouts. We found that planning
shape has a profound impact on the efficacy of Dyna for both perfect and
learned models. In addition to these findings regarding Dyna in general, our
results represent, to our knowledge, the first time that a learned dynamics
model has been successfully used for planning in the ALE, suggesting that Dyna
may be a viable approach to MBRL in the ALE and other high-dimensional
problems.
