  Policy gradient reinforcement learning (RL) algorithms have achieved
impressive performance in challenging learning tasks such as continuous
control, but suffer from high sample complexity. Experience replay is a
commonly used approach to improve sample efficiency, but gradient estimators
using past trajectories typically have high variance. Existing sampling
strategies for experience replay like uniform sampling or prioritised
experience replay do not explicitly try to control the variance of the gradient
estimates. In this paper, we propose an online learning algorithm, adaptive
experience selection (AES), to adaptively learn an experience sampling
distribution that explicitly minimises this variance. Using a regret
minimisation approach, AES iteratively updates the experience sampling
distribution to match the performance of a competitor distribution assumed to
have optimal variance. Sample non-stationarity is addressed by proposing a
dynamic (i.e. time changing) competitor distribution for which a closed-form
solution is proposed. We demonstrate that AES is a low-regret algorithm with
reasonable sample complexity. Empirically, AES has been implemented for deep
deterministic policy gradient and soft actor critic algorithms, and tested on 8
continuous control tasks from the OpenAI Gym library. Ours results show that
AES leads to significantly improved performance compared to currently available
experience sampling strategies for policy gradient.
