  Constraint-based offline reinforcement learning (RL) involves policy
constraints or imposing penalties on the value function to mitigate
overestimation errors caused by distributional shift. This paper focuses on a
limitation in existing offline RL methods with penalized value function,
indicating the potential for underestimation bias due to unnecessary bias
introduced in the value function. To address this concern, we propose
Exclusively Penalized Q-learning (EPQ), which reduces estimation bias in the
value function by selectively penalizing states that are prone to inducing
estimation errors. Numerical results show that our method significantly reduces
underestimation bias and improves performance in various offline control tasks
compared to other offline RL methods
