  Long-term engagement is preferred over immediate engagement in sequential
recommendation as it directly affects product operational metrics such as daily
active users (DAUs) and dwell time. Meanwhile, reinforcement learning (RL) is
widely regarded as a promising framework for optimizing long-term engagement in
sequential recommendation. However, due to expensive online interactions, it is
very difficult for RL algorithms to perform state-action value estimation,
exploration and feature extraction when optimizing long-term engagement. In
this paper, we propose ResAct which seeks a policy that is close to, but better
than, the online-serving policy. In this way, we can collect sufficient data
near the learned policy so that state-action values can be properly estimated,
and there is no need to perform online exploration. ResAct optimizes the policy
by first reconstructing the online behaviors and then improving it via a
Residual Actor. To extract long-term information, ResAct utilizes two
information-theoretical regularizers to confirm the expressiveness and
conciseness of features. We conduct experiments on a benchmark dataset and a
large-scale industrial dataset which consists of tens of millions of
recommendation requests. Experimental results show that our method
significantly outperforms the state-of-the-art baselines in various long-term
engagement optimization tasks.
