  It is a popular belief that model-based Reinforcement Learning (RL) is more
sample efficient than model-free RL, but in practice, it is not always true due
to overweighed model errors. In complex and noisy settings, model-based RL
tends to have trouble using the model if it does not know when to trust the
model.
  In this work, we find that better model usage can make a huge difference. We
show theoretically that if the use of model-generated data is restricted to
state-action pairs where the model error is small, the performance gap between
model and real rollouts can be reduced. It motivates us to use model rollouts
only when the model is confident about its predictions. We propose Masked
Model-based Actor-Critic (M2AC), a novel policy optimization algorithm that
maximizes a model-based lower-bound of the true value function. M2AC implements
a masking mechanism based on the model's uncertainty to decide whether its
prediction should be used or not. Consequently, the new algorithm tends to give
robust policy improvements. Experiments on continuous control benchmarks
demonstrate that M2AC has strong performance even when using long model
rollouts in very noisy environments, and it significantly outperforms previous
state-of-the-art methods.
