  Existing studies on constrained reinforcement learning (RL) may obtain a
well-performing policy in the training environment. However, when deployed in a
real environment, it may easily violate constraints that were originally
satisfied during training because there might be model mismatch between the
training and real environments. To address the above challenge, we formulate
the problem as constrained RL under model uncertainty, where the goal is to
learn a good policy that optimizes the reward and at the same time satisfy the
constraint under model mismatch. We develop a Robust Constrained Policy
Optimization (RCPO) algorithm, which is the first algorithm that applies to
large/continuous state space and has theoretical guarantees on worst-case
reward improvement and constraint violation at each iteration during the
training. We demonstrate the effectiveness of our algorithm on a set of RL
tasks with constraints.
