  Safety and robustness are two desired properties for any reinforcement
learning algorithm. CMDPs can handle additional safety constraints and RMDPs
can perform well under model uncertainties. In this paper, we propose to unite
these two frameworks resulting in robust constrained MDPs (RCMDPs). The
motivation is to develop a framework that can satisfy safety constraints while
also simultaneously offer robustness to model uncertainties. We develop the
RCMDP objective, derive gradient update formula to optimize this objective and
then propose policy gradient based algorithms. We also independently propose
Lyapunov based reward shaping for RCMDPs, yielding better stability and
convergence properties.
