  Since the objective functions of reinforcement learning problems are
typically highly nonconvex, it is desirable that policy gradient, the most
popular algorithm, escapes saddle points and arrives at second-order stationary
points. Existing results only consider vanilla policy gradient algorithms with
unbiased gradient estimators, but practical implementations under the
infinite-horizon discounted reward setting are biased due to finite-horizon
sampling. Moreover, actor-critic methods, whose second-order convergence has
not yet been established, are also biased due to the critic approximation of
the value function. We provide a novel second-order analysis of biased policy
gradient methods, including the vanilla gradient estimator computed from
Monte-Carlo sampling of trajectories as well as the double-loop actor-critic
algorithm, where in the inner loop the critic improves the approximation of the
value function via TD(0) learning. Separately, we also establish the
convergence of TD(0) on Markov chains irrespective of initial state
distribution.
