  The majority of current object detectors lack context: class predictions are
made independently from other detections. We propose to incorporate context in
object detection by post-processing the output of an arbitrary detector to
rescore the confidences of its detections. Rescoring is done by conditioning on
contextual information from the entire set of detections: their confidences,
predicted classes, and positions. We show that AP can be improved by simply
reassigning the detection confidence values such that true positives that
survive longer (i.e., those with the correct class and large IoU) are scored
higher than false positives or detections with small IoU. In this setting, we
use a bidirectional RNN with attention for contextual rescoring and introduce a
training target that uses the IoU with ground truth to maximize AP for the
given set of detections. The fact that our approach does not require access to
visual features makes it computationally inexpensive and agnostic to the
detection architecture. In spite of this simplicity, our model consistently
improves AP over strong pre-trained baselines (Cascade R-CNN and Faster R-CNN
with several backbones), particularly by reducing the confidence of duplicate
detections (a learned form of non-maximum suppression) and removing
out-of-context objects by conditioning on the confidences, classes, positions,
and sizes of the co-occurrent detections. Code is available at
https://github.com/LourencoVazPato/seeing-without-looking/
