  Scene graph generation (SGG) is built on top of detected objects to predict
object pairwise visual relations for describing the image content abstraction.
Existing works have revealed that if the links between objects are given as
prior knowledge, the performance of SGG is significantly improved. Inspired by
this observation, in this article, we propose a relation regularized network
(R2-Net), which can predict whether there is a relationship between two objects
and encode this relation into object feature refinement and better SGG.
Specifically, we first construct an affinity matrix among detected objects to
represent the probability of a relationship between two objects. Graph
convolution networks (GCNs) over this relation affinity matrix are then used as
object encoders, producing relation-regularized representations of objects.
With these relation-regularized features, our R2-Net can effectively refine
object labels and generate scene graphs. Extensive experiments are conducted on
the visual genome dataset for three SGG tasks (i.e., predicate classification,
scene graph classification, and scene graph detection), demonstrating the
effectiveness of our proposed method. Ablation studies also verify the key
roles of our proposed components in performance improvement.
