  Direct policy optimization in reinforcement learning is usually solved with
policy-gradient algorithms, which optimize policy parameters via stochastic
gradient ascent. This paper provides a new theoretical interpretation and
justification of these algorithms. First, we formulate direct policy
optimization in the optimization by continuation framework. The latter is a
framework for optimizing nonconvex functions where a sequence of surrogate
objective functions, called continuations, are locally optimized. Second, we
show that optimizing affine Gaussian policies and performing entropy
regularization can be interpreted as implicitly optimizing deterministic
policies by continuation. Based on these theoretical results, we argue that
exploration in policy-gradient algorithms consists in computing a continuation
of the return of the policy at hand, and that the variance of policies should
be history-dependent functions adapted to avoid local extrema rather than to
maximize the return of the policy.
