  Trust region policy optimization (TRPO) is a popular and empirically
successful policy search algorithm in Reinforcement Learning (RL) in which a
surrogate problem, that restricts consecutive policies to be 'close' to one
another, is iteratively solved. Nevertheless, TRPO has been considered a
heuristic algorithm inspired by Conservative Policy Iteration (CPI). We show
that the adaptive scaling mechanism used in TRPO is in fact the natural "RL
version" of traditional trust-region methods from convex analysis. We first
analyze TRPO in the planning setting, in which we have access to the model and
the entire state space. Then, we consider sample-based TRPO and establish
$\tilde O(1/\sqrt{N})$ convergence rate to the global optimum. Importantly, the
adaptive scaling mechanism allows us to analyze TRPO in regularized MDPs for
which we prove fast rates of $\tilde O(1/N)$, much like results in convex
optimization. This is the first result in RL of better rates when regularizing
the instantaneous cost or reward.
