  Probing is a popular method to discern what linguistic information is
contained in the representations of pre-trained language models. However, the
mechanism of selecting the probe model has recently been subject to intense
debate, as it is not clear if the probes are merely extracting information or
modeling the linguistic property themselves. To address this challenge, this
paper introduces a novel model-free approach to probing, by formulating probing
as a prompting task. We conduct experiments on five probing tasks and show that
our approach is comparable or better at extracting information than diagnostic
probes while learning much less on its own. We further combine the probing via
prompting approach with attention head pruning to analyze where the model
stores the linguistic information in its architecture. We then examine the
usefulness of a specific linguistic property for pre-training by removing the
heads that are essential to that property and evaluating the resulting model's
performance on language modeling.
