  The task of scene graph generation entails identifying object entities and
their corresponding interaction predicates in a given image (or video). Due to
the combinatorially large solution space, existing approaches to scene graph
generation assume certain factorization of the joint distribution to make the
estimation feasible (e.g., assuming that objects are conditionally independent
of predicate predictions). However, this fixed factorization is not ideal under
all scenarios (e.g., for images where an object entailed in interaction is
small and not discernible on its own). In this work, we propose a novel
framework for scene graph generation that addresses this limitation, as well as
introduces dynamic conditioning on the image, using message passing in a Markov
Random Field. This is implemented as an iterative refinement procedure wherein
each modification is conditioned on the graph generated in the previous
iteration. This conditioning across refinement steps allows joint reasoning
over entities and relations. This framework is realized via a novel and
end-to-end trainable transformer-based architecture. In addition, the proposed
framework can improve existing approach performance. Through extensive
experiments on Visual Genome and Action Genome benchmark datasets we show
improved performance on the scene graph generation.
