  In-context learning (ICL) is an appealing approach for semantic parsing due
to its few-shot nature and improved generalization. However, learning to parse
to rare domain-specific languages (DSLs) from just a few demonstrations is
challenging, limiting the performance of even the most capable LLMs. In this
work, we improve the effectiveness of ICL for semantic parsing by (1) using
general-purpose programming languages such as Python instead of DSLs, and (2)
augmenting prompts with a structured domain description that includes, e.g.,
the available classes and functions. We show that both these changes
significantly improve accuracy across three popular datasets. Combined, they
lead to dramatic improvements (e.g. 7.9% to 66.5% on SMCalFlow compositional
split), nearly closing the performance gap between easier i.i.d.\ and harder
compositional splits when used with a strong model, and reducing the need for a
large number of demonstrations. We find that the resemblance of the target
parse language to general-purpose code is a more important factor than the
language's popularity in pre-training corpora. Our findings provide an improved
methodology for building semantic parsers in the modern context of ICL with
LLMs.
