  Compositional generalization benchmarks for semantic parsing seek to assess
whether models can accurately compute meanings for novel sentences, but
operationalize this in terms of logical form (LF) prediction. This raises the
concern that semantically irrelevant details of the chosen LFs could shape
model performance. We argue that this concern is realized for the COGS
benchmark. COGS poses generalization splits that appear impossible for
present-day models, which could be taken as an indictment of those models.
However, we show that the negative results trace to incidental features of COGS
LFs. Converting these LFs to semantically equivalent ones and factoring out
capabilities unrelated to semantic interpretation, we find that even baseline
models get traction. A recent variable-free translation of COGS LFs suggests
similar conclusions, but we observe this format is not semantically equivalent;
it is incapable of accurately representing some COGS meanings. These findings
inform our proposal for ReCOGS, a modified version of COGS that comes closer to
assessing the target semantic capabilities while remaining very challenging.
Overall, our results reaffirm the importance of compositional generalization
and careful benchmark task design.
