  Sequential recommendation aims to capture users' dynamic interest and
predicts the next item of users' preference. Most sequential recommendation
methods use a deep neural network as sequence encoder to generate user and item
representations. Existing works mainly center upon designing a stronger
sequence encoder. However, few attempts have been made with training an
ensemble of networks as sequence encoders, which is more powerful than a single
network because an ensemble of parallel networks can yield diverse prediction
results and hence better accuracy. In this paper, we present Ensemble Modeling
with contrastive Knowledge Distillation for sequential recommendation (EMKD).
Our framework adopts multiple parallel networks as an ensemble of sequence
encoders and recommends items based on the output distributions of all these
networks. To facilitate knowledge transfer between parallel networks, we
propose a novel contrastive knowledge distillation approach, which performs
knowledge transfer from the representation level via Intra-network Contrastive
Learning (ICL) and Cross-network Contrastive Learning (CCL), as well as
Knowledge Distillation (KD) from the logits level via minimizing the
Kullback-Leibler divergence between the output distributions of the teacher
network and the student network. To leverage contextual information, we train
the primary masked item prediction task alongside the auxiliary attribute
prediction task as a multi-task learning scheme. Extensive experiments on
public benchmark datasets show that EMKD achieves a significant improvement
compared with the state-of-the-art methods. Besides, we demonstrate that our
ensemble method is a generalized approach that can also improve the performance
of other sequential recommenders. Our code is available at this link:
https://github.com/hw-du/EMKD.
