  In recommender systems such as news feed stream, it is essential to optimize
the long-term utilities in the continuous user-system interaction processes.
Previous works have proved the capability of reinforcement learning in this
problem. However, there are many practical challenges to implement deep
reinforcement learning in online systems, including low sample efficiency,
uncontrollable risks, and excessive variances. To address these issues, we
propose a novel reinforcement learning method, namely model-based
counterfactual advantage learning (MBCAL). The proposed method takes advantage
of the characteristics of recommender systems and draws ideas from the
model-based reinforcement learning method for higher sample efficiency. It has
two components: an environment model that predicts the instant user behavior
one-by-one in an auto-regressive form, and a future advantage model that
predicts the future utility. To alleviate the impact of excessive variance when
learning the future advantage model, we employ counterfactual comparisons
derived from the environment model. In consequence, the proposed method
possesses high sample efficiency and significantly lower variance; Also, it is
able to use existing user logs to avoid the risks of starting from scratch. In
contrast to its capability, its implementation cost is relatively low, which
fits well with practical systems. Theoretical analysis and elaborate
experiments are presented. Results show that the proposed method transcends the
other supervised learning and RL-based methods in both sample efficiency and
asymptotic performances.
