  Recently two approximate Newton methods were proposed for the optimisation of
Markov Decision Processes. While these methods were shown to have desirable
properties, such as a guarantee that the preconditioner is
negative-semidefinite when the policy is $\log$-concave with respect to the
policy parameters, and were demonstrated to have strong empirical performance
in challenging domains, such as the game of Tetris, no convergence analysis was
provided. The purpose of this paper is to provide such an analysis. We start by
providing a detailed analysis of the Hessian of a Markov Decision Process,
which is formed of a negative-semidefinite component, a positive-semidefinite
component and a remainder term. The first part of our analysis details how the
negative-semidefinite and positive-semidefinite components relate to each
other, and how these two terms contribute to the Hessian. The next part of our
analysis shows that under certain conditions, relating to the richness of the
policy class, the remainder term in the Hessian vanishes in the vicinity of a
local optimum. Finally, we bound the behaviour of this remainder term in terms
of the mixing time of the Markov chain induced by the policy parameters, where
this part of the analysis is applicable over the entire parameter space. Given
this analysis of the Hessian we then provide our local convergence analysis of
the approximate Newton framework.
