  State-of-the-art CNN based recognition models are often computationally
prohibitive to deploy on low-end devices. A promising high level approach
tackling this limitation is knowledge distillation, which let small student
model mimic cumbersome teacher model's output to get improved generalization.
However, related methods mainly focus on simple task of classification while do
not consider complex tasks like object detection. We show applying the vanilla
knowledge distillation to detection model gets minor gain. To address the
challenge of distilling knowledge in detection model, we propose a fine-grained
feature imitation method exploiting the cross-location discrepancy of feature
response. Our intuition is that detectors care more about local near object
regions. Thus the discrepancy of feature response on the near object anchor
locations reveals important information of how teacher model tends to
generalize. We design a novel mechanism to estimate those locations and let
student model imitate the teacher on them to get enhanced performance. We first
validate the idea on a developed lightweight toy detector which carries
simplest notion of current state-of-the-art anchor based detection models on
challenging KITTI dataset, our method generates up to 15% boost of mAP for the
student model compared to the non-imitated counterpart. We then extensively
evaluate the method with Faster R-CNN model under various scenarios with common
object detection benchmark of Pascal VOC and COCO, imitation alleviates up to
74% performance drop of student model compared to teacher. Codes released at
https://github.com/twangnh/Distilling-Object-Detectors
