  Two main challenges in Reinforcement Learning (RL) are designing appropriate
reward functions and ensuring the safety of the learned policy. To address
these challenges, we present a theoretical framework for Inverse Reinforcement
Learning (IRL) in constrained Markov decision processes. From a convex-analytic
perspective, we extend prior results on reward identifiability and
generalizability to both the constrained setting and a more general class of
regularizations. In particular, we show that identifiability up to potential
shaping (Cao et al., 2021) is a consequence of entropy regularization and may
generally no longer hold for other regularizations or in the presence of safety
constraints. We also show that to ensure generalizability to new transition
laws and constraints, the true reward must be identified up to a constant.
Additionally, we derive a finite sample guarantee for the suboptimality of the
learned rewards, and validate our results in a gridworld environment.
