  Reinforcement learning has seen increasing applications in real-world
contexts over the past few years. However, physical environments are often
imperfect and policies that perform well in simulation might not achieve the
same performance when applied elsewhere. A common approach to combat this is to
train agents in the presence of an adversary. An adversary acts to destabilise
the agent, which learns a more robust policy and can better handle realistic
conditions. Many real-world applications of reinforcement learning also make
use of goal-conditioning: this is particularly useful in the context of
robotics, as it allows the agent to act differently, depending on which goal is
selected. Here, we focus on the problem of goal-conditioned learning in the
presence of an adversary. We first present DigitFlip and CLEVR-Play, two novel
goal-conditioned environments that support acting against an adversary. Next,
we propose EHER and CHER -- two HER-based algorithms for goal-conditioned
learning -- and evaluate their performance. Finally, we unify the two threads
and introduce IGOAL: a novel framework for goal-conditioned learning in the
presence of an adversary. Experimental results show that combining IGOAL with
EHER allows agents to significantly outperform existing approaches, when acting
against both random and competent adversaries.
