  We study a class of constrained reinforcement learning (RL) problems in which
multiple constraint specifications are not identified before training. It is
challenging to identify appropriate constraint specifications due to the
undefined trade-off between the reward maximization objective and the
constraint satisfaction, which is ubiquitous in constrained decision-making. To
tackle this issue, we propose a new constrained RL approach that searches for
policy and constraint specifications together. This method features the
adaptation of relaxing the constraint according to a relaxation cost introduced
in the learning objective. Since this feature mimics how ecological systems
adapt to disruptions by altering operation, our approach is termed as resilient
constrained RL. Specifically, we provide a set of sufficient conditions that
balance the constraint satisfaction and the reward maximization in notion of
resilient equilibrium, propose a tractable formulation of resilient constrained
policy optimization that takes this equilibrium as an optimal solution, and
advocate two resilient constrained policy search algorithms with non-asymptotic
convergence guarantees on the optimality gap and constraint satisfaction.
Furthermore, we demonstrate the merits and the effectiveness of our approach in
computational experiments.
