  Many imitation learning (IL) algorithms employ inverse reinforcement learning
(IRL) to infer the intrinsic reward function that an expert is implicitly
optimizing for based on their demonstrated behaviors. However, in practice,
IRL-based IL can fail to accomplish the underlying task due to a misalignment
between the inferred reward and the objective of the task. In this paper, we
address the susceptibility of IL to such misalignment by introducing a
semi-supervised reward design paradigm called Protagonist Antagonist Guided
Adversarial Reward (PAGAR). PAGAR-based IL trains a policy to perform well
under mixed reward functions instead of a single reward function as in
IRL-based IL. We identify the theoretical conditions under which PAGAR-based IL
can avoid the task failures caused by reward misalignment. We also present a
practical on-and-off policy approach to implementing PAGAR-based IL.
Experimental results show that our algorithm outperforms standard IL baselines
in complex tasks and challenging transfer settings.
