  Off-policy deep reinforcement learning algorithms commonly compensate for
overestimation bias during temporal-difference learning by utilizing
pessimistic estimates of the expected target returns. In this work, we propose
Generalized Pessimism Learning (GPL), a strategy employing a novel learnable
penalty to enact such pessimism. In particular, we propose to learn this
penalty alongside the critic with dual TD-learning, a new procedure to estimate
and minimize the magnitude of the target returns bias with trivial
computational cost. GPL enables us to accurately counteract overestimation bias
throughout training without incurring the downsides of overly pessimistic
targets. By integrating GPL with popular off-policy algorithms, we achieve
state-of-the-art results in both competitive proprioceptive and pixel-based
benchmarks.
