  Model-based reinforcement learning is one approach to increase sample
efficiency. However, the accuracy of the dynamics model and the resulting
compounding error over modelled trajectories are commonly regarded as key
limitations. A natural question to ask is: How much more sample efficiency can
be gained by improving the learned dynamics models? Our paper empirically
answers this question for the class of model-based value expansion methods in
continuous control problems. Value expansion methods should benefit from
increased model accuracy by enabling longer rollout horizons and better value
function approximations. Our empirical study, which leverages oracle dynamics
models to avoid compounding model errors, shows that (1) longer horizons
increase sample efficiency, but the gain in improvement decreases with each
additional expansion step, and (2) the increased model accuracy only marginally
increases the sample efficiency compared to learned models with identical
horizons. Therefore, longer horizons and increased model accuracy yield
diminishing returns in terms of sample efficiency. These improvements in sample
efficiency are particularly disappointing when compared to model-free value
expansion methods. Even though they introduce no computational overhead, we
find their performance to be on-par with model-based value expansion methods.
Therefore, we conclude that the limitation of model-based value expansion
methods is not the model accuracy of the learned models. While higher model
accuracy is beneficial, our experiments show that even a perfect model will not
provide an un-rivalled sample efficiency but that the bottleneck lies
elsewhere.
