  The standard formulation of Reinforcement Learning lacks a practical way of
specifying what are admissible and forbidden behaviors. Most often,
practitioners go about the task of behavior specification by manually
engineering the reward function, a counter-intuitive process that requires
several iterations and is prone to reward hacking by the agent. In this work,
we argue that constrained RL, which has almost exclusively been used for safe
RL, also has the potential to significantly reduce the amount of work spent for
reward specification in applied RL projects. To this end, we propose to specify
behavioral preferences in the CMDP framework and to use Lagrangian methods to
automatically weigh each of these behavioral constraints. Specifically, we
investigate how CMDPs can be adapted to solve goal-based tasks while adhering
to several constraints simultaneously. We evaluate this framework on a set of
continuous control tasks relevant to the application of Reinforcement Learning
for NPC design in video games.
