  Adversarial Imitation Learning (AIL) is a class of popular state-of-the-art
Imitation Learning algorithms commonly used in robotics. In AIL, an artificial
adversary's misclassification is used as a reward signal that is optimized by
any standard Reinforcement Learning (RL) algorithm. Unlike most RL settings,
the reward in AIL is $differentiable$ but current model-free RL algorithms do
not make use of this property to train a policy. The reward is AIL is also
shaped since it comes from an adversary. We leverage the differentiability
property of the shaped AIL reward function and formulate a class of Actor
Residual Critic (ARC) RL algorithms. ARC algorithms draw a parallel to the
standard Actor-Critic (AC) algorithms in RL literature and uses a residual
critic, $C$ function (instead of the standard $Q$ function) to approximate only
the discounted future return (excluding the immediate reward). ARC algorithms
have similar convergence properties as the standard AC algorithms with the
additional advantage that the gradient through the immediate reward is exact.
For the discrete (tabular) case with finite states, actions, and known
dynamics, we prove that policy iteration with $C$ function converges to an
optimal policy. In the continuous case with function approximation and unknown
dynamics, we experimentally show that ARC aided AIL outperforms standard AIL in
simulated continuous-control and real robotic manipulation tasks. ARC
algorithms are simple to implement and can be incorporated into any existing
AIL implementation with an AC algorithm. Video and link to code are available
at: https://sites.google.com/view/actor-residual-critic.
