  Structured representations like graphs and parse trees play a crucial role in
many Natural Language Processing systems. In recent years, the advancements in
multi-turn user interfaces necessitate the need for controlling and updating
these structured representations given new sources of information. Although
there have been many efforts focusing on improving the performance of the
parsers that map text to graphs or parse trees, very few have explored the
problem of directly manipulating these representations. In this paper, we
explore the novel problem of graph modification, where the systems need to
learn how to update an existing scene graph given a new user's command. Our
novel models based on graph-based sparse transformer and cross attention
information fusion outperform previous systems adapted from the machine
translation and graph generation literature. We further contribute our large
graph modification datasets to the research community to encourage future
research for this new problem.
