  Policy gradients methods apply to complex, poorly understood, control
problems by performing stochastic gradient descent over a parameterized class
of polices. Unfortunately, even for simple control problems solvable by
standard dynamic programming techniques, policy gradient algorithms face
non-convex optimization problems and are widely understood to converge only to
a stationary point. This work identifies structural properties -- shared by
several classic control problems -- that ensure the policy gradient objective
function has no suboptimal stationary points despite being non-convex. When
these conditions are strengthened, this objective satisfies a
Polyak-lojasiewicz (gradient dominance) condition that yields convergence
rates. We also provide bounds on the optimality gap of any stationary point
when some of these conditions are relaxed.
