  Regularized reinforcement learning (RL), particularly the entropy-regularized
kind, has gained traction in optimal control and inverse RL. While standard
unregularized RL methods remain unaffected by changes in the number of actions,
we show that it can severely impact their regularized counterparts. This paper
demonstrates the importance of decoupling the regularizer from the action
space: that is, to maintain a consistent level of regularization regardless of
how many actions are involved to avoid over-regularization. Whereas the problem
can be avoided by introducing a task-specific temperature parameter, it is
often undesirable and cannot solve the problem when action spaces are
state-dependent. In the state-dependent action context, different states with
varying action spaces are regularized inconsistently. We introduce two
solutions: a static temperature selection approach and a dynamic counterpart,
universally applicable where this problem arises. Implementing these changes
improves performance on the DeepMind control suite in static and dynamic
temperature regimes and a biological sequence design task.
