  Deep learning has achieved remarkable successes in solving challenging
reinforcement learning (RL) problems when dense reward function is provided.
However, in sparse reward environment it still often suffers from the need to
carefully shape reward function to guide policy optimization. This limits the
applicability of RL in the real world since both reinforcement learning and
domain-specific knowledge are required. It is therefore of great practical
importance to develop algorithms which can learn from a binary signal
indicating successful task completion or other unshaped, sparse reward signals.
We propose a novel method called competitive experience replay, which
efficiently supplements a sparse reward by placing learning in the context of
an exploration competition between a pair of agents. Our method complements the
recently proposed hindsight experience replay (HER) by inducing an automatic
exploratory curriculum. We evaluate our approach on the tasks of reaching
various goal locations in an ant maze and manipulating objects with a robotic
arm. Each task provides only binary rewards indicating whether or not the goal
is achieved. Our method asymmetrically augments these sparse rewards for a pair
of agents each learning the same task, creating a competitive game designed to
drive exploration. Extensive experiments demonstrate that this method leads to
faster converge and improved task performance.
