  Current advances in recommender systems have been remarkably successful in
optimizing immediate engagement. However, long-term user engagement, a more
desirable performance metric, remains difficult to improve. Meanwhile, recent
reinforcement learning (RL) algorithms have shown their effectiveness in a
variety of long-term goal optimization tasks. For this reason, RL is widely
considered as a promising framework for optimizing long-term user engagement in
recommendation. Though promising, the application of RL heavily relies on
well-designed rewards, but designing rewards related to long-term user
engagement is quite difficult. To mitigate the problem, we propose a novel
paradigm, recommender systems with human preferences (or Preference-based
Recommender systems), which allows RL recommender systems to learn from
preferences about users historical behaviors rather than explicitly defined
rewards. Such preferences are easily accessible through techniques such as
crowdsourcing, as they do not require any expert knowledge. With PrefRec, we
can fully exploit the advantages of RL in optimizing long-term goals, while
avoiding complex reward engineering. PrefRec uses the preferences to
automatically train a reward function in an end-to-end manner. The reward
function is then used to generate learning signals to train the recommendation
policy. Furthermore, we design an effective optimization method for PrefRec,
which uses an additional value function, expectile regression and reward model
pre-training to improve the performance. We conduct experiments on a variety of
long-term user engagement optimization tasks. The results show that PrefRec
significantly outperforms previous state-of-the-art methods in all the tasks.
