  In recent years, deep off-policy actor-critic algorithms have become a
dominant approach to reinforcement learning for continuous control. One of the
primary drivers of this improved performance is the use of pessimistic value
updates to address function approximation errors, which previously led to
disappointing performance. However, a direct consequence of pessimism is
reduced exploration, running counter to theoretical support for the efficacy of
optimism in the face of uncertainty. So which approach is best? In this work,
we show that the most effective degree of optimism can vary both across tasks
and over the course of learning. Inspired by this insight, we introduce a novel
deep actor-critic framework, Tactical Optimistic and Pessimistic (TOP)
estimation, which switches between optimistic and pessimistic value learning
online. This is achieved by formulating the selection as a multi-arm bandit
problem. We show in a series of continuous control tasks that TOP outperforms
existing methods which rely on a fixed degree of optimism, setting a new state
of the art in challenging pixel-based environments. Since our changes are
simple to implement, we believe these insights can easily be incorporated into
a multitude of off-policy algorithms.
