  Model-based reinforcement learning has drawn considerable interest in recent
years, given its promise to improve sample efficiency. Moreover, when using
deep-learned models, it is potentially possible to learn compact models from
complex sensor data. However, the effectiveness of these learned models,
particularly their capacity to plan, i.e., to improve the current policy,
remains unclear. In this work, we study MuZero, a well-known deep model-based
reinforcement learning algorithm, and explore how far it achieves its learning
objective of a value-equivalent model and how useful the learned models are for
policy improvement. Amongst various other insights, we conclude that the model
learned by MuZero cannot effectively generalize to evaluate unseen policies,
which limits the extent to which we can additionally improve the current policy
by planning with the model.
