  In existing works that learn representation for object detection, the
relationship between a candidate window and the ground truth bounding box of an
object is simplified by thresholding their overlap. This paper shows
information loss in this simplification and picks up the relative location/size
information discarded by thresholding. We propose a representation learning
pipeline to use the relationship as supervision for improving the learned
representation in object detection. Such relationship is not limited to object
of the target category, but also includes surrounding objects of other
categories. We show that image regions with multiple contexts and multiple
rotations are effective in capturing such relationship during the
representation learning process and in handling the semantic and visual
variation caused by different window-object configurations. Experimental
results show that the representation learned by our approach can improve the
object detection accuracy by 6.4% in mean average precision (mAP) on
ILSVRC2014. On the challenging ILSVRC2014 test dataset, 48.6% mAP is achieved
by our single model and it is the best among published results. On PASCAL VOC,
it outperforms the state-of-the-art result of Fast RCNN by 3.3% in absolute
mAP.
