  Pre-trained language models have achieved great success in various
large-scale information retrieval tasks. However, most of pretraining tasks are
based on counterfeit retrieval data where the query produced by the tailored
rule is assumed as the user's issued query on the given document or passage.
Therefore, we explore to use large-scale click logs to pretrain a language
model instead of replying on the simulated queries. Specifically, we propose to
use user behavior features to pretrain a debiased language model for document
ranking. Extensive experiments on Baidu desensitization click logs validate the
effectiveness of our method. Our team on WSDM Cup 2023 Pre-training for Web
Search won the 1st place with a Discounted Cumulative Gain @ 10 (DCG@10) score
of 12.16525 on the final leaderboard.
