  Recent advancements in pre-trained language models (PLMs) have demonstrated
that these models possess some degree of syntactic awareness. To leverage this
knowledge, we propose a novel chart-based method for extracting parse trees
from masked language models (LMs) without the need to train separate parsers.
Our method computes a score for each span based on the distortion of contextual
representations resulting from linguistic perturbations. We design a set of
perturbations motivated by the linguistic concept of constituency tests, and
use these to score each span by aggregating the distortion scores. To produce a
parse tree, we use chart parsing to find the tree with the minimum score. Our
method consistently outperforms previous state-of-the-art methods on English
with masked LMs, and also demonstrates superior performance in a multilingual
setting, outperforming the state of the art in 6 out of 8 languages. Notably,
although our method does not involve parameter updates or extensive
hyperparameter search, its performance can even surpass some unsupervised
parsing methods that require fine-tuning. Our analysis highlights that the
distortion of contextual representation resulting from syntactic perturbation
can serve as an effective indicator of constituency across languages.
