  Safe reinforcement learning (RL) studies problems where an intelligent agent
has to not only maximize reward but also avoid exploring unsafe areas. In this
study, we propose CUP, a novel policy optimization method based on Constrained
Update Projection framework that enjoys rigorous safety guarantee. Central to
our CUP development is the newly proposed surrogate functions along with the
performance bound. Compared to previous safe RL methods, CUP enjoys the
benefits of 1) CUP generalizes the surrogate functions to generalized advantage
estimator (GAE), leading to strong empirical performance. 2) CUP unifies
performance bounds, providing a better understanding and interpretability for
some existing algorithms; 3) CUP provides a non-convex implementation via only
first-order optimizers, which does not require any strong approximation on the
convexity of the objectives. To validate our CUP method, we compared CUP
against a comprehensive list of safe RL baselines on a wide range of tasks.
Experiments show the effectiveness of CUP both in terms of reward and safety
constraint satisfaction. We have opened the source code of CUP at this link
https://github.com/zmsn-2077/ CUP-safe-rl.
