  Prompt-based probing has been widely used in evaluating the abilities of
pretrained language models (PLMs). Unfortunately, recent studies have
discovered such an evaluation may be inaccurate, inconsistent and unreliable.
Furthermore, the lack of understanding its inner workings, combined with its
wide applicability, has the potential to lead to unforeseen risks for
evaluating and applying PLMs in real-world applications. To discover,
understand and quantify the risks, this paper investigates the prompt-based
probing from a causal view, highlights three critical biases which could induce
biased results and conclusions, and proposes to conduct debiasing via causal
intervention. This paper provides valuable insights for the design of unbiased
datasets, better probing frameworks and more reliable evaluations of pretrained
language models. Furthermore, our conclusions also echo that we need to rethink
the criteria for identifying better pretrained language models. We openly
released the source code and data at https://github.com/c-box/causalEval.
