  A widely-used actor-critic reinforcement learning algorithm for continuous
control, Deep Deterministic Policy Gradients (DDPG), suffers from the
overestimation problem, which can negatively affect the performance. Although
the state-of-the-art Twin Delayed Deep Deterministic Policy Gradient (TD3)
algorithm mitigates the overestimation issue, it can lead to a large
underestimation bias. In this paper, we propose to use the Boltzmann softmax
operator for value function estimation in continuous control. We first
theoretically analyze the softmax operator in continuous action space. Then, we
uncover an important property of the softmax operator in actor-critic
algorithms, i.e., it helps to smooth the optimization landscape, which sheds
new light on the benefits of the operator. We also design two new algorithms,
Softmax Deep Deterministic Policy Gradients (SD2) and Softmax Deep Double
Deterministic Policy Gradients (SD3), by building the softmax operator upon
single and double estimators, which can effectively improve the overestimation
and underestimation bias. We conduct extensive experiments on challenging
continuous control tasks, and results show that SD3 outperforms
state-of-the-art methods.
