  The goal of this paper is to detect objects by exploiting their
interrelationships. Contrary to existing methods, which learn objects and
relations separately, our key idea is to learn the object-relation distribution
jointly. We first propose a novel way of creating a graphical representation of
an image from inter-object relation priors and initial class predictions, we
call a context-likelihood graph. We then learn the joint distribution with an
energy-based modeling technique which allows to sample and refine the
context-likelihood graph iteratively for a given image. Our formulation of
jointly learning the distribution enables us to generate a more accurate graph
representation of an image which leads to a better object detection
performance. We demonstrate the benefits of our context-likelihood graph
formulation and the energy-based graph refinement via experiments on the Visual
Genome and MS-COCO datasets where we achieve a consistent improvement over
object detectors like DETR and Faster-RCNN, as well as alternative methods
modeling object interrelationships separately. Our method is detector agnostic,
end-to-end trainable, and especially beneficial for rare object classes.
