  We study robust reinforcement learning (RL) with the goal of determining a
well-performing policy that is robust against model mismatch between the
training simulator and the testing environment. Previous policy-based robust RL
algorithms mainly focus on the tabular setting under uncertainty sets that
facilitate robust policy evaluation, but are no longer tractable when the
number of states scales up. To this end, we propose two novel uncertainty set
formulations, one based on double sampling and the other on an integral
probability metric. Both make large-scale robust RL tractable even when one
only has access to a simulator. We propose a robust natural actor-critic (RNAC)
approach that incorporates the new uncertainty sets and employs function
approximation. We provide finite-time convergence guarantees for the proposed
RNAC algorithm to the optimal robust policy within the function approximation
error. Finally, we demonstrate the robust performance of the policy learned by
our proposed RNAC approach in multiple MuJoCo environments and a real-world
TurtleBot navigation task.
