  In machine learning systems, privileged features refer to the features that
are available during offline training but inaccessible for online serving.
Previous studies have recognized the importance of privileged features and
explored ways to tackle online-offline discrepancies. A typical practice is
privileged features distillation (PFD): train a teacher model using all
features (including privileged ones) and then distill the knowledge from the
teacher model using a student model (excluding the privileged features), which
is then employed for online serving. In practice, the pointwise cross-entropy
loss is often adopted for PFD. However, this loss is insufficient to distill
the ranking ability for CTR prediction. First, it does not consider the
non-i.i.d. characteristic of the data distribution, i.e., other items on the
same page significantly impact the click probability of the candidate item.
Second, it fails to consider the relative item order ranked by the teacher
model's predictions, which is essential to distill the ranking ability. To
address these issues, we first extend the pointwise-based PFD to the
listwise-based PFD. We then define the calibration-compatible property of
distillation loss and show that commonly used listwise losses do not satisfy
this property when employed as distillation loss, thus compromising the model's
calibration ability, which is another important measure for CTR prediction. To
tackle this dilemma, we propose Calibration-compatible LIstwise Distillation
(CLID), which employs carefully-designed listwise distillation loss to achieve
better ranking ability than the pointwise-based PFD while preserving the
model's calibration ability. We theoretically prove it is
calibration-compatible. Extensive experiments on public datasets and a
production dataset collected from the display advertising system of Alibaba
further demonstrate the effectiveness of CLID.
