  News recommendation is a widely adopted technique to provide personalized
news feeds for the user. Recently, pre-trained language models (PLMs) have
demonstrated the great capability of natural language understanding and
benefited news recommendation via improving news modeling. However, most
existing works simply finetune the PLM with the news recommendation task, which
may suffer from the known domain shift problem between the pre-training corpus
and downstream news texts. Moreover, PLMs usually contain a large volume of
parameters and have high computational overhead, which imposes a great burden
on low-latency online services. In this paper, we propose Tiny-NewsRec, which
can improve both the effectiveness and the efficiency of PLM-based news
recommendation. We first design a self-supervised domain-specific post-training
method to better adapt the general PLM to the news domain with a contrastive
matching task between news titles and news bodies. We further propose a
two-stage knowledge distillation method to improve the efficiency of the large
PLM-based news recommendation model while maintaining its performance. Multiple
teacher models originated from different time steps of our post-training
procedure are used to transfer comprehensive knowledge to the student in both
its post-training and finetuning stage. Extensive experiments on two real-world
datasets validate the effectiveness and efficiency of our method.
