  Deep Q Network (DQN) is a very successful algorithm, yet the inherent problem
of reinforcement learning, i.e. the exploit-explore balance, remains. In this
work, we introduce entropy regularization into DQN and propose SQN. We find
that the backup equation of soft Q learning can enjoy the corrective feedback
if we view the soft backup as policy improvement in the form of Q, instead of
policy evaluation. We show that Soft Q Learning with Corrective Feedback
(SQL-CF) underlies the on-plicy nature of SQL and the equivalence of SQL and
Soft Policy Gradient (SPG). With these insights, we propose an on-policy
version of deep Q learning algorithm, i.e. Q On-Policy (QOP). We experiment
with QOP on a self-play environment called Google Research Football (GRF). The
QOP algorithm exhibits great stability and efficiency in training GRF agents.
