  S{\o}gaard (2020) obtained results suggesting the fraction of trees occurring
in the test data isomorphic to trees in the training set accounts for a
non-trivial variation in parser performance. Similar to other statistical
analyses in NLP, the results were based on evaluating linear regressions.
However, the study had methodological issues and was undertaken using a small
sample size leading to unreliable results. We present a replication study in
which we also bin sentences by length and find that only a small subset of
sentences vary in performance with respect to graph isomorphism. Further, the
correlation observed between parser performance and graph isomorphism in the
wild disappears when controlling for covariants. However, in a controlled
experiment, where covariants are kept fixed, we do observe a strong
correlation. We suggest that conclusions drawn from statistical analyses like
this need to be tempered and that controlled experiments can complement them by
more readily teasing factors apart.
