  Experience replay plays a crucial role in improving the sample efficiency of
deep reinforcement learning agents. Recent advances in experience replay
propose using Mixup (Zhang et al., 2018) to further improve sample efficiency
via synthetic sample generation. We build upon this technique with Neighborhood
Mixup Experience Replay (NMER), a geometrically-grounded replay buffer that
interpolates transitions with their closest neighbors in state-action space.
NMER preserves a locally linear approximation of the transition manifold by
only applying Mixup between transitions with vicinal state-action features.
Under NMER, a given transition's set of state action neighbors is dynamic and
episode agnostic, in turn encouraging greater policy generalizability via
inter-episode interpolation. We combine our approach with recent off-policy
deep reinforcement learning algorithms and evaluate on continuous control
environments. We observe that NMER improves sample efficiency by an average 94%
(TD3) and 29% (SAC) over baseline replay buffers, enabling agents to
effectively recombine previous experiences and learn from limited data.
