  Syntax is a latent hierarchical structure which underpins the robust and
compositional nature of human language. In this work, we explore the hypothesis
that syntactic dependencies can be represented in language model attention
distributions and propose a new method to induce these structures
theory-agnostically. Instead of modeling syntactic relations as defined by
annotation schemata, we model a more general property implicit in the
definition of dependency relations, syntactic substitutability. This property
captures the fact that words at either end of a dependency can be substituted
with words from the same category. Substitutions can be used to generate a set
of syntactically invariant sentences whose representations are then used for
parsing. We show that increasing the number of substitutions used improves
parsing accuracy on natural data. On long-distance subject-verb agreement
constructions, our method achieves 79.5% recall compared to 8.9% using a
previous method. Our method also provides improvements when transferred to a
different parsing setup, demonstrating that it generalizes.
