  Scene graph prediction --- classifying the set of objects and predicates in a
visual scene --- requires substantial training data. However, most predicates
only occur a handful of times making them difficult to learn. We introduce the
first scene graph prediction model that supports few-shot learning of
predicates. Existing scene graph generation models represent objects using
pretrained object detectors or word embeddings that capture semantic object
information at the cost of encoding information about which relationships they
afford. So, these object representations are unable to generalize to new
few-shot relationships. We introduce a framework that induces object
representations that are structured according to their visual relationships.
Unlike past methods, our framework embeds objects that afford similar
relationships closer together. This property allows our model to perform well
in the few-shot setting. For example, applying the 'riding' predicate
transformation to 'person' modifies the representation towards objects like
'skateboard' and 'horse' that enable riding. We generate object representations
by learning predicates trained as message passing functions within a new graph
convolution framework. The object representations are used to build few-shot
predicate classifiers for rare predicates with as few as 1 labeled example. We
achieve a 5-shot performance of 22.70 recall@50, a 3.7 increase when compared
to strong transfer learning baselines.
