  Object detection involves two sub-tasks, i.e. localizing objects in an image
and classifying them into various categories. For existing CNN-based detectors,
we notice the widespread divergence between localization and classification,
which leads to degradation in performance. In this work, we propose a mutual
learning framework to modulate the two tasks. In particular, the two tasks are
forced to learn from each other with a novel mutual labeling strategy. Besides,
we introduce a simple yet effective IoU rescoring scheme, which further reduces
the divergence. Moreover, we define a Spearman rank correlation-based metric to
quantify the divergence, which correlates well with the detection performance.
The proposed approach is general-purpose and can be easily injected into
existing detectors such as FCOS and RetinaNet. We achieve a significant
performance gain over the baseline detectors on the COCO dataset.
