  Label assignment in object detection aims to assign targets, foreground or
background, to sampled regions in an image. Unlike labeling for image
classification, this problem is not well defined due to the object's bounding
box. In this paper, we investigate the problem from a perspective of
distillation, hence we call Label Assignment Distillation (LAD). Our initial
motivation is very simple, we use a teacher network to generate labels for the
student. This can be achieved in two ways: either using the teacher's
prediction as the direct targets (soft label), or through the hard labels
dynamically assigned by the teacher (LAD). Our experiments reveal that: (i) LAD
is more effective than soft-label, but they are complementary. (ii) Using LAD,
a smaller teacher can also improve a larger student significantly, while
soft-label can't. We then introduce Co-learning LAD, in which two networks
simultaneously learn from scratch and the role of teacher and student are
dynamically interchanged. Using PAA-ResNet50 as a teacher, our LAD techniques
can improve detectors PAA-ResNet101 and PAA-ResNeXt101 to $46 \rm AP$ and
$47.5\rm AP$ on the COCO test-dev set. With a stronger teacher PAA-SwinB, we
improve the students PAA-ResNet50 to $43.7\rm AP$ by only 1x schedule training
and standard setting, and PAA-ResNet101 to $47.9\rm AP$, significantly
surpassing the current methods. Our source code and checkpoints are released at
https://git.io/JrDZo.
