  Value-based deep Reinforcement Learning (RL) algorithms suffer from the
estimation bias primarily caused by function approximation and temporal
difference (TD) learning. This problem induces faulty state-action value
estimates and therefore harms the performance and robustness of the learning
algorithms. Although several techniques were proposed to tackle, learning
algorithms still suffer from this bias. Here, we introduce a technique that
eliminates the estimation bias in off-policy continuous control algorithms
using the experience replay mechanism. We adaptively learn the weighting
hyper-parameter beta in the Weighted Twin Delayed Deep Deterministic Policy
Gradient algorithm. Our method is named Adaptive-WD3 (AWD3). We show through
continuous control environments of OpenAI gym that our algorithm matches or
outperforms the state-of-the-art off-policy policy gradient learning
algorithms.
