  Long-term user engagement (LTE) optimization in sequential recommender
systems (SRS) is shown to be suited by reinforcement learning (RL) which finds
a policy to maximize long-term rewards. Meanwhile, RL has its shortcomings,
particularly requiring a large number of online samples for exploration, which
is risky in real-world applications. One of the appealing ways to avoid the
risk is to build a simulator and learn the optimal recommendation policy in the
simulator. In LTE optimization, the simulator is to simulate multiple users'
daily feedback for given recommendations. However, building a user simulator
with no reality-gap, i.e., can predict user's feedback exactly, is unrealistic
because the users' reaction patterns are complex and historical logs for each
user are limited, which might mislead the simulator-based recommendation
policy. In this paper, we present a practical simulator-based recommender
policy training approach, Simulation-to-Recommendation (Sim2Rec) to handle the
reality-gap problem for LTE optimization. Specifically, Sim2Rec introduces a
simulator set to generate various possibilities of user behavior patterns, then
trains an environment-parameter extractor to recognize users' behavior patterns
in the simulators. Finally, a context-aware policy is trained to make the
optimal decisions on all of the variants of the users based on the inferred
environment-parameters. The policy is transferable to unseen environments
(e.g., the real world) directly as it has learned to recognize all various user
behavior patterns and to make the correct decisions based on the inferred
environment-parameters. Experiments are conducted in synthetic environments and
a real-world large-scale ride-hailing platform, DidiChuxing. The results show
that Sim2Rec achieves significant performance improvement, and produces robust
recommendations in unseen environments.
