  User modeling is critical for many personalized web services. Many existing
methods model users based on their behaviors and the labeled data of target
tasks. However, these methods cannot exploit useful information in unlabeled
user behavior data, and their performance may be not optimal when labeled data
is scarce. Motivated by pre-trained language models which are pre-trained on
large-scale unlabeled corpus to empower many downstream tasks, in this paper we
propose to pre-train user models from large-scale unlabeled user behaviors
data. We propose two self-supervision tasks for user model pre-training. The
first one is masked behavior prediction, which can model the relatedness
between historical behaviors. The second one is next $K$ behavior prediction,
which can model the relatedness between past and future behaviors. The
pre-trained user models are finetuned in downstream tasks to learn
task-specific user representations. Experimental results on two real-world
datasets validate the effectiveness of our proposed user model pre-training
method.
