  Compositionality -- the ability to combine familiar units like words into
novel phrases and sentences -- has been the focus of intense interest in
artificial intelligence in recent years. To test compositional generalization
in semantic parsing, Keysers et al. (2020) introduced Compositional Freebase
Queries (CFQ). This dataset maximizes the similarity between the test and train
distributions over primitive units, like words, while maximizing the compound
divergence: the dissimilarity between test and train distributions over larger
structures, like phrases. Dependency parsing, however, lacks a compositional
generalization benchmark. In this work, we introduce a gold-standard set of
dependency parses for CFQ, and use this to analyze the behavior of a
state-of-the art dependency parser (Qi et al., 2020) on the CFQ dataset. We
find that increasing compound divergence degrades dependency parsing
performance, although not as dramatically as semantic parsing performance.
Additionally, we find the performance of the dependency parser does not
uniformly degrade relative to compound divergence, and the parser performs
differently on different splits with the same compound divergence. We explore a
number of hypotheses for what causes the non-uniform degradation in dependency
parsing performance, and identify a number of syntactic structures that drive
the dependency parser's lower performance on the most challenging splits.
