  We observe that several existing policy gradient methods (such as vanilla
policy gradient, PPO, A2C) may suffer from overly large gradients when the
current policy is close to deterministic (even in some very simple
environments), leading to an unstable training process. To address this issue,
we propose a new method, called \emph{target distribution learning} (TDL), for
policy improvement in reinforcement learning. TDL alternates between proposing
a target distribution and training the policy network to approach the target
distribution. TDL is more effective in constraining the KL divergence between
updated policies, and hence leads to more stable policy improvements over
iterations. Our experiments show that TDL algorithms perform comparably to (or
better than) state-of-the-art algorithms for most continuous control tasks in
the MuJoCo environment while being more stable in training.
