  Applications based on image retrieval require editing and associating in
intermediate spaces that are representative of the high-level concepts like
objects and their relationships rather than dense, pixel-level representations
like RGB images or semantic-label maps. We focus on one such representation,
scene graphs, and propose a novel scene expansion task where we enrich an input
seed graph by adding new nodes (objects) and the corresponding relationships.
To this end, we formulate scene graph expansion as a sequential prediction task
involving multiple steps of first predicting a new node and then predicting the
set of relationships between the newly predicted node and previous nodes in the
graph. We propose a sequencing strategy for observed graphs that retains the
clustering patterns amongst nodes. In addition, we leverage external knowledge
to train our graph generation model, enabling greater generalization of node
predictions. Due to the inefficiency of existing maximum mean discrepancy (MMD)
based metrics for graph generation problems in evaluating predicted
relationships between nodes (objects), we design novel metrics that
comprehensively evaluate different aspects of predicted relations. We conduct
extensive experiments on Visual Genome and VRD datasets to evaluate the
expanded scene graphs using the standard MMD-based metrics and our proposed
metrics. We observe that the graphs generated by our method, GEMS, better
represent the real distribution of the scene graphs than the baseline methods
like GraphRNN.
