  In many real world applications, reinforcement learning agents have to
optimize multiple objectives while following certain rules or satisfying a list
of constraints. Classical methods based on reward shaping, i.e. a weighted
combination of different objectives in the reward signal, or Lagrangian
methods, including constraints in the loss function, have no guarantees that
the agent satisfies the constraints at all points in time and can lead to
undesired behavior. When a discrete policy is extracted from an action-value
function, safe actions can be ensured by restricting the action space at
maximization, but can lead to sub-optimal solutions among feasible
alternatives. In this work, we propose Constrained Q-learning, a novel
off-policy reinforcement learning framework restricting the action space
directly in the Q-update to learn the optimal Q-function for the induced
constrained MDP and the corresponding safe policy. In addition to single-step
constraints referring only to the next action, we introduce a formulation for
approximate multi-step constraints under the current target policy based on
truncated value-functions. We analyze the advantages of Constrained Q-learning
in the tabular case and compare Constrained DQN to reward shaping and
Lagrangian methods in the application of high-level decision making in
autonomous driving, considering constraints for safety, keeping right and
comfort. We train our agent in the open-source simulator SUMO and on the real
HighD data set.
