  Policy gradient algorithms have been widely applied to Markov decision
processes and reinforcement learning problems in recent years. Regularization
with various entropy functions is often used to encourage exploration and
improve stability. This paper proposes an approximate Newton method for the
policy gradient algorithm with entropy regularization. In the case of Shannon
entropy, the resulting algorithm reproduces the natural policy gradient
algorithm. For other entropy functions, this method results in brand-new policy
gradient algorithms. We prove that all these algorithms enjoy Newton-type
quadratic convergence and that the corresponding gradient flow converges
globally to the optimal solution. We use synthetic and industrial-scale
examples to demonstrate that the proposed approximate Newton method typically
converges in single-digit iterations, often orders of magnitude faster than
other state-of-the-art algorithms.
