  Object detection remains as one of the most notorious open problems in
computer vision. Despite large strides in accuracy in recent years, modern
object detectors have started to saturate on popular benchmarks raising the
question of how far we can reach with deep learning tools and tricks. Here, by
employing 2 state-of-the-art object detection benchmarks, and analyzing more
than 15 models over 4 large scale datasets, we I) carefully determine the upper
bound in AP, which is 91.6% on VOC (test2007), 78.2% on COCO (val2017), and
58.9% on OpenImages V4 (validation), regardless of the IOU threshold. These
numbers are much better than the mAP of the best model (47.9% on VOC, and 46.9%
on COCO; IOUs=.5:.05:.95), II) characterize the sources of errors in object
detectors, in a novel and intuitive way, and find that classification error
(confusion with other classes and misses) explains the largest fraction of
errors and weighs more than localization and duplicate errors, and III) analyze
the invariance properties of models when surrounding context of an object is
removed, when an object is placed in an incongruent background, and when images
are blurred or flipped vertically. We find that models generate a lot of boxes
on empty regions and that context is more important for detecting small objects
than larger ones. Our work taps into the tight relationship between object
detection and object recognition and offers insights for building better
models. Our code is publicly available at
https://github.com/aliborji/Deetctionupper bound.git.
