  Scene graphs provide a rich, structured representation of a scene by encoding
the entities (objects) and their spatial relationships in a graphical format.
This representation has proven useful in several tasks, such as question
answering, captioning, and even object detection, to name a few. Current
approaches take a generation-by-classification approach where the scene graph
is generated through labeling of all possible edges between objects in a scene,
which adds computational overhead to the approach. This work introduces a
generative transformer-based approach to generating scene graphs beyond link
prediction. Using two transformer-based components, we first sample a possible
scene graph structure from detected objects and their visual features. We then
perform predicate classification on the sampled edges to generate the final
scene graph. This approach allows us to efficiently generate scene graphs from
images with minimal inference overhead. Extensive experiments on the Visual
Genome dataset demonstrate the efficiency of the proposed approach. Without
bells and whistles, we obtain, on average, 20.7% mean recall (mR@100) across
different settings for scene graph generation (SGG), outperforming
state-of-the-art SGG approaches while offering competitive performance to
unbiased SGG approaches.
