  This paper studies the constrained/safe reinforcement learning (RL) problem
with sparse indicator signals for constraint violations. We propose a
model-based approach to enable RL agents to effectively explore the environment
with unknown system dynamics and environment constraints given a significantly
small number of violation budgets. We employ the neural network ensemble model
to estimate the prediction uncertainty and use model predictive control as the
basic control framework. We propose the robust cross-entropy method to optimize
the control sequence considering the model uncertainty and constraints. We
evaluate our methods in the Safety Gym environment. The results show that our
approach learns to complete the tasks with a much smaller number of constraint
violations than state-of-the-art baselines. Additionally, we are able to
achieve several orders of magnitude better sample efficiency when compared with
constrained model-free RL approaches. The code is available at
\url{https://github.com/liuzuxin/safe-mbrl}.
