  A significant challenge in reinforcement learning is quantifying the complex
relationship between actions and long-term rewards. The effects may manifest
themselves over a long sequence of state-action pairs, making them hard to
pinpoint. In this paper, we propose a method to link transitions with
significant deviations in state with unusually large variations in subsequent
rewards. Such transitions are marked as possible causal effects, and the
corresponding state-action pairs are added to a separate replay buffer. In
addition, we include \textit{contrastive} samples corresponding to transitions
from a similar state but with differing actions. Including this Contrastive
Experience Replay (CER) during training is shown to outperform standard
value-based methods on 2D navigation tasks. We believe that CER can be useful
for a broad class of learning tasks, including for any off-policy reinforcement
learning algorithm.
