  NLP models have progressed drastically in recent years, according to numerous
datasets proposed to evaluate performance. Questions remain, however, about how
particular dataset design choices may impact the conclusions we draw about
model capabilities. In this work, we investigate this question in the domain of
compositional generalization. We examine the performance of six modeling
approaches across 4 datasets, split according to 8 compositional splitting
strategies, ranking models by 18 compositional generalization splits in total.
Our results show that: i) the datasets, although all designed to evaluate
compositional generalization, rank modeling approaches differently; ii)
datasets generated by humans align better with each other than they with
synthetic datasets, or than synthetic datasets among themselves; iii)
generally, whether datasets are sampled from the same source is more predictive
of the resulting model ranking than whether they maintain the same
interpretation of compositionality; and iv) which lexical items are used in the
data can strongly impact conclusions. Overall, our results demonstrate that
much work remains to be done when it comes to assessing whether popular
evaluation datasets measure what they intend to measure, and suggest that
elucidating more rigorous standards for establishing the validity of evaluation
sets could benefit the field.
