  In value-based deep reinforcement learning methods, approximation of value
functions induces overestimation bias and leads to suboptimal policies. We show
that in deep actor-critic methods that aim to overcome the overestimation bias,
if the reinforcement signals received by the agent have a high variance, a
significant underestimation bias arises. To minimize the underestimation, we
introduce a parameter-free, novel deep Q-learning variant. Our Q-value update
rule combines the notions behind Clipped Double Q-learning and Maxmin
Q-learning by computing the critic objective through the nested combination of
maximum and minimum operators to bound the approximate value estimates. We
evaluate our modification on the suite of several OpenAI Gym continuous control
tasks, improving the state-of-the-art in every environment tested.
