  We propose a novel value approximation method, namely Eigensubspace
Regularized Critic (ERC) for deep reinforcement learning (RL). ERC is motivated
by an analysis of the dynamics of Q-value approximation error in the
Temporal-Difference (TD) method, which follows a path defined by the
1-eigensubspace of the transition kernel associated with the Markov Decision
Process (MDP). It reveals a fundamental property of TD learning that has
remained unused in previous deep RL approaches. In ERC, we propose a
regularizer that guides the approximation error tending towards the
1-eigensubspace, resulting in a more efficient and stable path of value
approximation. Moreover, we theoretically prove the convergence of the ERC
method. Besides, theoretical analysis and experiments demonstrate that ERC
effectively reduces the variance of value functions. Among 26 tasks in the
DMControl benchmark, ERC outperforms state-of-the-art methods for 20. Besides,
it shows significant advantages in Q-value approximation and variance
reduction. Our code is available at https://sites.google.com/view/erc-ecml23/.
