  Visual relationship detection is fundamental for holistic image
understanding. However, the localization and classification of (subject,
predicate, object) triplets remain challenging tasks, due to the combinatorial
explosion of possible relationships, their long-tailed distribution in natural
images, and an expensive annotation process. This paper introduces a novel
weakly-supervised method for visual relationship detection that relies on
minimal image-level predicate labels. A graph neural network is trained to
classify predicates in images from a graph representation of detected objects,
implicitly encoding an inductive bias for pairwise relations. We then frame
relationship detection as the explanation of such a predicate classifier, i.e.
we obtain a complete relation by recovering the subject and object of a
predicted predicate. We present results comparable to recent fully- and
weakly-supervised methods on three diverse and challenging datasets: HICO-DET
for human-object interaction, Visual Relationship Detection for generic
object-to-object relations, and UnRel for unusual triplets; demonstrating
robustness to non-comprehensive annotations and good few-shot generalization.
