  Approximation of the value functions in value-based deep reinforcement
learning induces overestimation bias, resulting in suboptimal policies. We show
that when the reinforcement signals received by the agents have a high
variance, deep actor-critic approaches that overcome the overestimation bias
lead to a substantial underestimation bias. We first address the detrimental
issues in the existing approaches that aim to overcome such underestimation
error. Then, through extensive statistical analysis, we introduce a novel,
parameter-free Deep Q-learning variant to reduce this underestimation bias in
deterministic policy gradients. By sampling the weights of a linear combination
of two approximate critics from a highly shrunk estimation bias interval, our
Q-value update rule is not affected by the variance of the rewards received by
the agents throughout learning. We test the performance of the introduced
improvement on a set of MuJoCo and Box2D continuous control tasks and
demonstrate that it considerably outperforms the existing approaches and
improves the state-of-the-art by a significant margin.
