  Reinforcement learning with sparse rewards is challenging because an agent
can rarely obtain non-zero rewards and hence, gradient-based optimization of
parameterized policies can be incremental and slow. Recent work demonstrated
that using a memory buffer of previous successful trajectories can result in
more effective policies. However, existing methods may overly exploit past
successful experiences, which can encourage the agent to adopt sub-optimal and
myopic behaviors. In this work, instead of focusing on good experiences with
limited diversity, we propose to learn a trajectory-conditioned policy to
follow and expand diverse past trajectories from a memory buffer. Our method
allows the agent to reach diverse regions in the state space and improve upon
the past trajectories to reach new states. We empirically show that our
approach significantly outperforms count-based exploration methods (parametric
approach) and self-imitation learning (parametric approach with non-parametric
memory) on various complex tasks with local optima. In particular, without
using expert demonstrations or resetting to arbitrary states, we achieve the
state-of-the-art scores under five billion number of frames, on challenging
Atari games such as Montezuma's Revenge and Pitfall.
