  In this paper, we explore how to use a small amount of new data to update a
task-oriented semantic parsing model when the desired output for some examples
has changed. When making updates in this way, one potential problem that arises
is the presence of conflicting data, or out-of-date labels in the original
training set. To evaluate the impact of this understudied problem, we propose
an experimental setup for simulating changes to a neural semantic parser. We
show that the presence of conflicting data greatly hinders learning of an
update, then explore several methods to mitigate its effect. Our multi-task and
data selection methods lead to large improvements in model accuracy compared to
a naive data-mixing strategy, and our best method closes 86% of the accuracy
gap between this baseline and an oracle upper bound.
