  This paper presents a model-free approximation for the Hessian of the
performance of deterministic policies to use in the context of Reinforcement
Learning based on Quasi-Newton steps in the policy parameters. We show that the
approximate Hessian converges to the exact Hessian at the optimal policy, and
allows for a superlinear convergence in the learning, provided that the policy
parametrization is rich. The natural policy gradient method can be interpreted
as a particular case of the proposed method. We analytically verify the
formulation in a simple linear case and compare the convergence of the proposed
method with the natural policy gradient in a nonlinear example.
