  A widely-studied deep reinforcement learning (RL) technique known as
Prioritized Experience Replay (PER) allows agents to learn from transitions
sampled with non-uniform probability proportional to their temporal-difference
(TD) error. Although it has been shown that PER is one of the most crucial
components for the overall performance of deep RL methods in discrete action
domains, many empirical studies indicate that it considerably underperforms
actor-critic algorithms in continuous control. We theoretically show that actor
networks cannot be effectively trained with transitions that have large TD
errors. As a result, the approximate policy gradient computed under the
Q-network diverges from the actual gradient computed under the optimal
Q-function. Motivated by this, we introduce a novel experience replay sampling
framework for actor-critic methods, which also regards issues with stability
and recent findings behind the poor empirical performance of PER. The
introduced algorithm suggests a new branch of improvements to PER and schedules
effective and efficient training for both actor and critic networks. An
extensive set of experiments verifies our theoretical claims and demonstrates
that the introduced method significantly outperforms the competing approaches
and obtains state-of-the-art results over the standard off-policy actor-critic
algorithms.
