  Policy gradient methods with actor-critic schemes demonstrate tremendous
empirical successes, especially when the actors and critics are parameterized
by neural networks. However, it remains less clear whether such "neural" policy
gradient methods converge to globally optimal policies and whether they even
converge at all. We answer both the questions affirmatively in the
overparameterized regime. In detail, we prove that neural natural policy
gradient converges to a globally optimal policy at a sublinear rate. Also, we
show that neural vanilla policy gradient converges sublinearly to a stationary
point. Meanwhile, by relating the suboptimality of the stationary points to the
representation power of neural actor and critic classes, we prove the global
optimality of all stationary points under mild regularity conditions.
Particularly, we show that a key to the global optimality and convergence is
the "compatibility" between the actor and critic, which is ensured by sharing
neural architectures and random initializations across the actor and critic. To
the best of our knowledge, our analysis establishes the first global optimality
and convergence guarantees for neural policy gradient methods.
