  Meta-reinforcement learning has widely been used as a learning-to-learn
framework to solve unseen tasks with limited experience. However, the aspect of
constraint violations has not been adequately addressed in the existing works,
making their application restricted in real-world settings. In this paper, we
study the problem of meta-safe reinforcement learning (Meta-SRL) through the
CMDP-within-online framework to establish the first provable guarantees in this
important setting. We obtain task-averaged regret bounds for the reward
maximization (optimality gap) and constraint violations using gradient-based
meta-learning and show that the task-averaged optimality gap and constraint
satisfaction improve with task-similarity in a static environment or
task-relatedness in a dynamic environment. Several technical challenges arise
when making this framework practical. To this end, we propose a meta-algorithm
that performs inexact online learning on the upper bounds of within-task
optimality gap and constraint violations estimated by off-policy stationary
distribution corrections. Furthermore, we enable the learning rates to be
adapted for every task and extend our approach to settings with a competing
dynamically changing oracle. Finally, experiments are conducted to demonstrate
the effectiveness of our approach.
