  Recently, increasing efforts have been focused on Weakly Supervised Scene
Graph Generation (WSSGG). The mainstream solution for WSSGG typically follows
the same pipeline: they first align text entities in the weak image-level
supervisions (e.g., unlocalized relation triplets or captions) with image
regions, and then train SGG models in a fully-supervised manner with aligned
instance-level "pseudo" labels. However, we argue that most existing WSSGG
works only focus on object-consistency, which means the grounded regions should
have the same object category label as text entities. While they neglect
another basic requirement for an ideal alignment: interaction-consistency,
which means the grounded region pairs should have the same interactions (i.e.,
visual relations) as text entity pairs. Hence, in this paper, we propose to
enhance a simple grounding module with both object-aware and interaction-aware
knowledge to acquire more reliable pseudo labels. To better leverage these two
types of knowledge, we regard them as two teachers and fuse their generated
targets to guide the training process of our grounding module. Specifically, we
design two different strategies to adaptively assign weights to different
teachers by assessing their reliability on each training sample. Extensive
experiments have demonstrated that our method consistently improves WSSGG
performance on various kinds of weak supervision.
