  In this work, we investigate the application of Taylor expansions in
reinforcement learning. In particular, we propose Taylor expansion policy
optimization, a policy optimization formalism that generalizes prior work
(e.g., TRPO) as a first-order special case. We also show that Taylor expansions
intimately relate to off-policy evaluation. Finally, we show that this new
formulation entails modifications which improve the performance of several
state-of-the-art distributed algorithms.
