  Trust-region methods based on Kullback-Leibler divergence are pervasively
used to stabilize policy optimization in reinforcement learning. In this paper,
we exploit more flexible metrics and examine two natural extensions of policy
optimization with Wasserstein and Sinkhorn trust regions, namely Wasserstein
policy optimization (WPO) and Sinkhorn policy optimization (SPO). Instead of
restricting the policy to a parametric distribution class, we directly optimize
the policy distribution and derive their closed-form policy updates based on
the Lagrangian duality. Theoretically, we show that WPO guarantees a monotonic
performance improvement, and SPO provably converges to WPO as the entropic
regularizer diminishes. Moreover, we prove that with a decaying Lagrangian
multiplier to the trust region constraint, both methods converge to global
optimality. Experiments across tabular domains, robotic locomotion, and
continuous control tasks further demonstrate the performance improvement of
both approaches, more robustness of WPO to sample insufficiency, and faster
convergence of SPO, over state-of-art policy gradient methods.
