  Despite the frequent challenges posed by ambiguity when representing meaning
via natural language, it is often ignored or deliberately removed in tasks
mapping language to formally-designed representations, which generally assume a
one-to-one mapping between linguistic and formal representations. We attempt to
address this shortcoming by introducing AmP, a framework, dataset, and
challenge for translating ambiguous natural language to formal representations
like logic and code. We define templates and generate data for five
well-documented linguistic ambiguities. Using AmP, we investigate how several
few-shot text-to-code systems handle ambiguity, introducing three new metrics.
We find that large pre-trained models perform poorly at capturing the
distribution of possible meanings without deliberate instruction. However,
models are able to capture the distribution well when ambiguity is attested in
their inputs. These results motivate a call for including ambiguity explicitly
in datasets and promote considering the distribution of possible outputs when
evaluating systems. Data and code: https://github.com/esteng/ambiguous_parsing
