  Model-free reinforcement learning (RL) requires a large number of trials to
learn a good policy, especially in environments with sparse rewards. We explore
a method to improve the sample efficiency when we have access to
demonstrations. Our approach, Backplay, uses a single demonstration to
construct a curriculum for a given task. Rather than starting each training
episode in the environment's fixed initial state, we start the agent near the
end of the demonstration and move the starting point backwards during the
course of training until we reach the initial state. Our contributions are that
we analytically characterize the types of environments where Backplay can
improve training speed, demonstrate the effectiveness of Backplay both in large
grid worlds and a complex four player zero-sum game (Pommerman), and show that
Backplay compares favorably to other competitive methods known to improve
sample efficiency. This includes reward shaping, behavioral cloning, and
reverse curriculum generation.
