  Entropy regularization is an important idea in reinforcement learning, with
great success in recent algorithms like Soft Q Network (SQN) and Soft
Actor-Critic (SAC1). In this work, we extend this idea into the on-policy
realm. We propose the soft policy gradient theorem (SPGT) for on-policy maximum
entropy reinforcement learning. With SPGT, a series of new policy optimization
algorithms are derived, such as SPG, SA2C, SA3C, SDDPG, STRPO, SPPO, SIMPALA
and so on. We find that SDDPG is equivalent to SAC1. For policy gradient, the
policy network is often represented as a Gaussian distribution with a global
action variance, which damages the representation capacity. We introduce a
local action variance for policy network and find it can work collaboratively
with the idea of entropy regularization. Our method outperforms prior works on
a range of benchmark tasks. Furthermore, our method can be easily extended to
large scale experiment with great stability and parallelism.
