  Recent studies have proposed unified user modeling frameworks that leverage
user behavior data from various applications. Many of them benefit from
utilizing users' behavior sequences as plain texts, representing rich
information in any domain or system without losing generality. Hence, a
question arises: Can language modeling for user history corpus help improve
recommender systems? While its versatile usability has been widely investigated
in many domains, its applications to recommender systems still remain
underexplored. We show that language modeling applied directly to task-specific
user histories achieves excellent results on diverse recommendation tasks.
Also, leveraging additional task-agnostic user histories delivers significant
performance benefits. We further demonstrate that our approach can provide
promising transfer learning capabilities for a broad spectrum of real-world
recommender systems, even on unseen domains and services.
