  Recovering reward function from expert demonstrations is a fundamental
problem in reinforcement learning. The recovered reward function captures the
motivation of the expert. Agents can imitate experts by following these reward
functions in their environment, which is known as apprentice learning. However,
the agents may face environments different from the demonstrations, and
therefore, desire transferable reward functions. Classical reward learning
methods such as inverse reinforcement learning (IRL) or, equivalently,
adversarial imitation learning (AIL), recover reward functions coupled with
training dynamics, which are hard to be transferable. Previous
dynamics-agnostic reward learning methods rely on assumptions such as that the
reward function has to be state-only, restricting their applicability. In this
work, we present a dynamics-agnostic discriminator-ensemble reward learning
method (DARL) within the AIL framework, capable of learning both state-action
and state-only reward functions. DARL achieves this by decoupling the reward
function from training dynamics, employing a dynamics-agnostic discriminator on
a latent space derived from the original state-action space. This latent space
is optimized to minimize information on the dynamics. We moreover discover the
policy-dependency issue of the AIL framework that reduces the transferability.
DARL represents the reward function as an ensemble of discriminators during
training to eliminate policy dependencies. Empirical studies on MuJoCo tasks
with changed dynamics show that DARL better recovers the reward function and
results in better imitation performance in transferred environments, handling
both state-only and state-action reward scenarios.
