  Scaling reinforcement learning (RL) to recommender systems (RS) is promising
since maximizing the expected cumulative rewards for RL agents meets the
objective of RS, i.e., improving customers' long-term satisfaction. A key
approach to this goal is offline RL, which aims to learn policies from logged
data. However, the high-dimensional action space and the non-stationary
dynamics in commercial RS intensify distributional shift issues, making it
challenging to apply offline RL methods to RS. To alleviate the action
distribution shift problem in extracting RL policy from static trajectories, we
propose Value Penalized Q-learning (VPQ), an uncertainty-based offline RL
algorithm. It penalizes the unstable Q-values in the regression target by
uncertainty-aware weights, without the need to estimate the behavior policy,
suitable for RS with a large number of items. We derive the penalty weights
from the variances across an ensemble of Q-functions. To alleviate
distributional shift issues at test time, we further introduce the critic
framework to integrate the proposed method with classic RS models. Extensive
experiments conducted on two real-world datasets show that the proposed method
could serve as a gain plugin for existing RS models.
