  With the thriving of pre-trained language model (PLM) widely verified in
various of NLP tasks, pioneer efforts attempt to explore the possible
cooperation of the general textual information in PLM with the personalized
behavioral information in user historical behavior sequences to enhance
sequential recommendation (SR). However, despite the commonalities of input
format and task goal, there are huge gaps between the behavioral and textual
information, which obstruct thoroughly modeling SR as language modeling via
PLM. To bridge the gap, we propose a novel Unified pre-trained language model
enhanced sequential recommendation (UPSR), aiming to build a unified
pre-trained recommendation model for multi-domain recommendation tasks. We
formally design five key indicators, namely naturalness, domain consistency,
informativeness, noise & ambiguity, and text length, to guide the text-item
adaptation and behavior sequence-text sequence adaptation differently for
pre-training and fine-tuning stages, which are essential but under-explored by
previous works. In experiments, we conduct extensive evaluations on seven
datasets with both tuning and zero-shot settings and achieve the overall best
performance. Comprehensive model analyses also provide valuable insights for
behavior modeling via PLM, shedding light on large pre-trained recommendation
models. The source codes will be released in the future.
