  Safe reinforcement learning is extremely challenging--not only must the agent
explore an unknown environment, it must do so while ensuring no safety
constraint violations. We formulate this safe reinforcement learning (RL)
problem using the framework of a finite-horizon Constrained Markov Decision
Process (CMDP) with an unknown transition probability function, where we model
the safety requirements as constraints on the expected cumulative costs that
must be satisfied during all episodes of learning. We propose a model-based
safe RL algorithm that we call Doubly Optimistic and Pessimistic Exploration
(DOPE), and show that it achieves an objective regret
$\tilde{O}(|\mathcal{S}|\sqrt{|\mathcal{A}| K})$ without violating the safety
constraints during learning, where $|\mathcal{S}|$ is the number of states,
$|\mathcal{A}|$ is the number of actions, and $K$ is the number of learning
episodes. Our key idea is to combine a reward bonus for exploration (optimism)
with a conservative constraint (pessimism), in addition to the standard
optimistic model-based exploration. DOPE is not only able to improve the
objective regret bound, but also shows a significant empirical performance
improvement as compared to earlier optimism-pessimism approaches.
