  Experience replay is one of the most commonly used approaches to improve the
sample efficiency of reinforcement learning algorithms. In this work, we
propose an approach to select and replay sequences of transitions in order to
accelerate the learning of a reinforcement learning agent in an off-policy
setting. In addition to selecting appropriate sequences, we also artificially
construct transition sequences using information gathered from previous
agent-environment interactions. These sequences, when replayed, allow value
function information to trickle down to larger sections of the
state/state-action space, thereby making the most of the agent's experience. We
demonstrate our approach on modified versions of standard reinforcement
learning tasks such as the mountain car and puddle world problems and
empirically show that it enables better learning of value functions as compared
to other forms of experience replay. Further, we briefly discuss some of the
possible extensions to this work, as well as applications and situations where
this approach could be particularly useful.
