  A scene graph is a semantic representation that expresses the objects,
attributes, and relationships between objects in a scene. Scene graphs play an
important role in many cross modality tasks, as they are able to capture the
interactions between images and texts. In this paper, we focus on scene graph
modification (SGM), where the system is required to learn how to update an
existing scene graph based on a natural language query. Unlike previous
approaches that rebuilt the entire scene graph, we frame SGM as a graph
expansion task by introducing the incremental structure expanding (ISE). ISE
constructs the target graph by incrementally expanding the source graph without
changing the unmodified structure. Based on ISE, we further propose a model
that iterates between nodes prediction and edges prediction, inferring more
accurate and harmonious expansion decisions progressively. In addition, we
construct a challenging dataset that contains more complicated queries and
larger scene graphs than existing datasets. Experiments on four benchmarks
demonstrate the effectiveness of our approach, which surpasses the previous
state-of-the-art model by large margins.
