  The theoretical advantages of distributional reinforcement learning~(RL) over
classical RL remain elusive despite its remarkable empirical performance.
Starting from Categorical Distributional RL~(CDRL), we attribute the potential
superiority of distributional RL to a derived distribution-matching
regularization by applying a return density function decomposition technique.
This unexplored regularization in the distributional RL context is aimed at
capturing additional return distribution information regardless of only its
expectation, contributing to an augmented reward signal in the policy
optimization. Compared with the entropy regularization in MaxEnt RL that
explicitly optimizes the policy to encourage the exploration, the resulting
regularization in CDRL implicitly optimizes policies guided by the new reward
signal to align with the uncertainty of target return distributions, leading to
an uncertainty-aware exploration effect. Finally, extensive experiments
substantiate the importance of this uncertainty-aware regularization in
distributional RL on the empirical benefits over classical RL.
