  Transformer-based retrieval and reranking models for text document search are
often refined through knowledge distillation together with contrastive
learning. A tight distribution matching between the teacher and student models
can be hard as over-calibration may degrade training effectiveness when a
teacher does not perform well. This paper contrastively reweights KL divergence
terms to prioritize the alignment between a student and a teacher model for
proper separation of positive and negative documents. This paper analyzes and
evaluates the proposed loss function on the MS MARCO and BEIR datasets to
demonstrate its effectiveness in improving the relevance of tested student
models.
