  Reinforcement Learning (RL) agents in the real world must satisfy safety
constraints in addition to maximizing a reward objective. Model-based RL
algorithms hold promise for reducing unsafe real-world actions: they may
synthesize policies that obey all constraints using simulated samples from a
learned model. However, imperfect models can result in real-world constraint
violations even for actions that are predicted to satisfy all constraints. We
propose Conservative and Adaptive Penalty (CAP), a model-based safe RL
framework that accounts for potential modeling errors by capturing model
uncertainty and adaptively exploiting it to balance the reward and the cost
objectives. First, CAP inflates predicted costs using an uncertainty-based
penalty. Theoretically, we show that policies that satisfy this conservative
cost constraint are guaranteed to also be feasible in the true environment. We
further show that this guarantees the safety of all intermediate solutions
during RL training. Further, CAP adaptively tunes this penalty during training
using true cost feedback from the environment. We evaluate this conservative
and adaptive penalty-based approach for model-based safe RL extensively on
state and image-based environments. Our results demonstrate substantial gains
in sample-efficiency while incurring fewer violations than prior safe RL
algorithms. Code is available at: https://github.com/Redrew/CAP
