  Imitation learning is a class of promising policy learning algorithms that is
free from many practical issues with reinforcement learning, such as the reward
design issue and the exploration hardness. However, the current imitation
algorithm struggles to achieve both high performance and high in-environment
sample efficiency simultaneously. Behavioral Cloning (BC) does not need
in-environment interactions, but it suffers from the covariate shift problem
which harms its performance. Adversarial Imitation Learning (AIL) turns
imitation learning into a distribution matching problem. It can achieve better
performance on some tasks but it requires a large number of in-environment
interactions. Inspired by the recent success of EfficientZero in RL, we propose
EfficientImitate (EI), a planning-based imitation learning method that can
achieve high in-environment sample efficiency and performance simultaneously.
Our algorithmic contribution in this paper is two-fold. First, we extend AIL
into the MCTS-based RL. Second, we show the seemingly incompatible two classes
of imitation algorithms (BC and AIL) can be naturally unified under our
framework, enjoying the benefits of both. We benchmark our method not only on
the state-based DeepMind Control Suite, but also on the image version which
many previous works find highly challenging. Experimental results show that EI
achieves state-of-the-art results in performance and sample efficiency. EI
shows over 4x gain in performance in the limited sample setting on state-based
and image-based tasks and can solve challenging problems like Humanoid, where
previous methods fail with small amount of interactions. Our code is available
at https://github.com/zhaohengyin/EfficientImitate.
