  Model-Based Reinforcement Learning involves learning a \textit{dynamics
model} from data, and then using this model to optimise behaviour, most often
with an online \textit{planner}. Much of the recent research along these lines
presents a particular set of design choices, involving problem definition,
model learning and planning. Given the multiple contributions, it is difficult
to evaluate the effects of each. This paper sets out to disambiguate the role
of different design choices for learning dynamics models, by comparing their
performance to planning with a ground-truth model -- the simulator. First, we
collect a rich dataset from the training sequence of a model-free agent on 5
domains of the DeepMind Control Suite. Second, we train feed-forward dynamics
models in a supervised fashion, and evaluate planner performance while varying
and analysing different model design choices, including ensembling,
stochasticity, multi-step training and timestep size. Besides the quantitative
analysis, we describe a set of qualitative findings, rules of thumb, and future
research directions for planning with learned dynamics models. Videos of the
results are available at https://sites.google.com/view/learning-better-models.
