  Transformer-based detectors (DETRs) are becoming popular for their simple
framework, but the large model size and heavy time consumption hinder their
deployment in the real world. While knowledge distillation (KD) can be an
appealing technique to compress giant detectors into small ones for comparable
detection performance and low inference cost. Since DETRs formulate object
detection as a set prediction problem, existing KD methods designed for classic
convolution-based detectors may not be directly applicable. In this paper, we
propose DETRDistill, a novel knowledge distillation method dedicated to
DETR-families. Specifically, we first design a Hungarian-matching logits
distillation to encourage the student model to have the exact predictions as
that of teacher DETRs. Next, we propose a target-aware feature distillation to
help the student model learn from the object-centric features of the teacher
model. Finally, in order to improve the convergence rate of the student DETR,
we introduce a query-prior assignment distillation to speed up the student
model learning from well-trained queries and stable assignment of the teacher
model. Extensive experimental results on the COCO dataset validate the
effectiveness of our approach. Notably, DETRDistill consistently improves
various DETRs by more than 2.0 mAP, even surpassing their teacher models.
