  In recent years, fully differentiable rigid body physics simulators have been
developed, which can be used to simulate a wide range of robotic systems. In
the context of reinforcement learning for control, these simulators
theoretically allow algorithms to be applied directly to analytic gradients of
the reward function. However, to date, these gradients have proved extremely
challenging to use, and are outclassed by algorithms using no gradient
information at all. In this work we present a novel algorithm, cross entropy
analytic policy gradients, that is able to leverage these gradients to
outperform state of art deep reinforcement learning on a set of challenging
nonlinear control problems.
