  Imitation learning in a high-dimensional environment is challenging. Most
inverse reinforcement learning (IRL) methods fail to outperform the
demonstrator in such a high-dimensional environment, e.g., Atari domain. To
address this challenge, we propose a novel reward learning module to generate
intrinsic reward signals via a generative model. Our generative method can
perform better forward state transition and backward action encoding, which
improves the module's dynamics modeling ability in the environment. Thus, our
module provides the imitation agent both the intrinsic intention of the
demonstrator and a better exploration ability, which is critical for the agent
to outperform the demonstrator. Empirical results show that our method
outperforms state-of-the-art IRL methods on multiple Atari games, even with
one-life demonstration. Remarkably, our method achieves performance that is up
to 5 times the performance of the demonstration.
