  This paper proposes model-free imitation learning named Entropy-Regularized
Imitation Learning (ERIL) that minimizes the reverse Kullback-Leibler (KL)
divergence. ERIL combines forward and inverse reinforcement learning (RL) under
the framework of an entropy-regularized Markov decision process. An inverse RL
step computes the log-ratio between two distributions by evaluating two binary
discriminators. The first discriminator distinguishes the state generated by
the forward RL step from the expert's state. The second discriminator, which is
structured by the theory of entropy regularization, distinguishes the
state-action-next-state tuples generated by the learner from the expert ones.
One notable feature is that the second discriminator shares hyperparameters
with the forward RL, which can be used to control the discriminator's ability.
A forward RL step minimizes the reverse KL estimated by the inverse RL step. We
show that minimizing the reverse KL divergence is equivalent to finding an
optimal policy. Our experimental results on MuJoCo-simulated environments and
vision-based reaching tasks with a robotic arm show that ERIL is more
sample-efficient than the baseline methods. We apply the method to human
behaviors that perform a pole-balancing task and describe how the estimated
reward functions show how every subject achieves her goal.
