  DETR is a novel end-to-end transformer architecture object detector, which
significantly outperforms classic detectors when scaling up the model size. In
this paper, we focus on the compression of DETR with knowledge distillation.
While knowledge distillation has been well-studied in classic detectors, there
is a lack of researches on how to make it work effectively on DETR. We first
provide experimental and theoretical analysis to point out that the main
challenge in DETR distillation is the lack of consistent distillation points.
Distillation points refer to the corresponding inputs of the predictions for
student to mimic, and reliable distillation requires sufficient distillation
points which are consistent between teacher and student. Based on this
observation, we propose a general knowledge distillation paradigm for
DETR(KD-DETR) with consistent distillation points sampling. Specifically, we
decouple detection and distillation tasks by introducing a set of specialized
object queries to construct distillation points. In this paradigm, we further
propose a general-to-specific distillation points sampling strategy to explore
the extensibility of KD-DETR. Extensive experiments on different DETR
architectures with various scales of backbones and transformer layers validate
the effectiveness and generalization of KD-DETR. KD-DETR boosts the performance
of DAB-DETR with ResNet-18 and ResNet-50 backbone to 41.4$\%$, 45.7$\%$ mAP,
respectively, which are 5.2$\%$, 3.5$\%$ higher than the baseline, and
ResNet-50 even surpasses the teacher model by $2.2\%$.
