  Model-based reinforcement learning (MBRL) aims to learn a dynamic model to
reduce the number of interactions with real-world environments. However, due to
estimation error, rollouts in the learned model, especially those of long
horizons, fail to match the ones in real-world environments. This mismatching
has seriously impacted the sample complexity of MBRL. The phenomenon can be
attributed to the fact that previous works employ supervised learning to learn
the one-step transition models, which has inherent difficulty ensuring the
matching of distributions from multi-step rollouts. Based on the claim, we
propose to learn the transition model by matching the distributions of
multi-step rollouts sampled from the transition model and the real ones via
WGAN. We theoretically show that matching the two can minimize the difference
of cumulative rewards between the real transition and the learned one. Our
experiments also show that the proposed Model Imitation method can compete or
outperform the state-of-the-art in terms of sample complexity and average
return.
