  Experience replay, which enables the agents to remember and reuse experience
from the past, has played a significant role in the success of off-policy
reinforcement learning (RL). To utilize the experience replay efficiently, the
existing sampling methods allow selecting out more meaningful experiences by
imposing priorities on them based on certain metrics (e.g. TD-error). However,
they may result in sampling highly biased, redundant transitions since they
compute the sampling rate for each transition independently, without
consideration of its importance in relation to other transitions. In this
paper, we aim to address the issue by proposing a new learning-based sampling
method that can compute the relative importance of transition. To this end, we
design a novel permutation-equivariant neural architecture that takes contexts
from not only features of each transition (local) but also those of others
(global) as inputs. We validate our framework, which we refer to as Neural
Experience Replay Sampler (NERS), on multiple benchmark tasks for both
continuous and discrete control tasks and show that it can significantly
improve the performance of various off-policy RL methods. Further analysis
confirms that the improvements of the sample efficiency indeed are due to
sampling diverse and meaningful transitions by NERS that considers both local
and global contexts.
