  In reinforcement learning, the advantage function is critical for policy
improvement, but is often extracted from a learned Q-function. A natural
question is: Why not learn the advantage function directly? In this work, we
introduce VA-learning, which directly learns advantage function and value
function using bootstrapping, without explicit reference to Q-functions.
VA-learning learns off-policy and enjoys similar theoretical guarantees as
Q-learning. Thanks to the direct learning of advantage function and value
function, VA-learning improves the sample efficiency over Q-learning both in
tabular implementations and deep RL agents on Atari-57 games. We also identify
a close connection between VA-learning and the dueling architecture, which
partially explains why a simple architectural change to DQN agents tends to
improve performance.
