  We contribute to micro-data model-based reinforcement learning (MBRL) by
rigorously comparing popular generative models using a fixed (random shooting)
control agent. We find that on an environment that requires multimodal
posterior predictives, mixture density nets outperform all other models by a
large margin. When multimodality is not required, our surprising finding is
that we do not need probabilistic posterior predictives: deterministic models
are on par, in fact they consistently (although non-significantly) outperform
their probabilistic counterparts. We also found that heteroscedasticity at
training time, perhaps acting as a regularizer, improves predictions at longer
horizons. At the methodological side, we design metrics and an experimental
protocol which can be used to evaluate the various models, predicting their
asymptotic performance when using them on the control problem. Using this
framework, we improve the state-of-the-art sample complexity of MBRL on Acrobot
by two to four folds, using an aggressive training schedule which is outside of
the hyperparameter interval usually considered
