  Actor-critic methods solve reinforcement learning problems by updating a
parameterized policy known as an actor in a direction that increases an
estimate of the expected return known as a critic. However, existing
actor-critic methods only use values or gradients of the critic to update the
policy parameter. In this paper, we propose a novel actor-critic method called
the guide actor-critic (GAC). GAC firstly learns a guide actor that locally
maximizes the critic and then it updates the policy parameter based on the
guide actor by supervised learning. Our main theoretical contributions are two
folds. First, we show that GAC updates the guide actor by performing
second-order optimization in the action space where the curvature matrix is
based on the Hessians of the critic. Second, we show that the deterministic
policy gradient method is a special case of GAC when the Hessians are ignored.
Through experiments, we show that our method is a promising reinforcement
learning method for continuous controls.
