  Q-learning with value function approximation may have the poor performance
because of overestimation bias and imprecise estimate. Specifically,
overestimation bias is from the maximum operator over noise estimate, which is
exaggerated using the estimate of a subsequent state. Inspired by the recent
advance of deep reinforcement learning and Double Q-learning, we introduce the
decorrelated double Q-learning (D2Q). Specifically, we introduce the
decorrelated regularization item to reduce the correlation between value
function approximators, which can lead to less biased estimation and low
variance. The experimental results on a suite of MuJoCo continuous control
tasks demonstrate that our decorrelated double Q-learning can effectively
improve the performance.
