  In-context learning has shown great success in i.i.d semantic parsing splits,
where the training and test sets are drawn from the same distribution. In this
setup, models are typically prompted with demonstrations that are similar to
the input utterance. However, in the setup of compositional generalization,
where models are tested on outputs with structures that are absent from the
training set, selecting similar demonstrations is insufficient, as often no
example will be similar enough to the input. In this work, we propose a method
to select diverse demonstrations that aims to collectively cover all of the
structures required in the output program, in order to encourage the model to
generalize to new structures from these demonstrations. We empirically show
that combining diverse demonstrations with in-context learning substantially
improves performance across three compositional generalization semantic parsing
datasets in the pure in-context learning setup and when combined with
finetuning.
