  Recent research shows that pre-trained language models (PLMs) suffer from
"prompt bias" in factual knowledge extraction, i.e., prompts tend to introduce
biases toward specific labels. Prompt bias presents a significant challenge in
assessing the factual knowledge within PLMs. Therefore, this paper aims to
improve the reliability of existing benchmarks by thoroughly investigating and
mitigating prompt bias. We show that: 1) all prompts in the experiments exhibit
non-negligible bias, with gradient-based prompts like AutoPrompt and OptiPrompt
displaying significantly higher levels of bias; 2) prompt bias can amplify
benchmark accuracy unreasonably by overfitting the test datasets, especially on
imbalanced datasets like LAMA. Based on these findings, we propose a
representation-based approach to mitigate the prompt bias during inference
time. Specifically, we first estimate the biased representation using
prompt-only querying, and then remove it from the model's internal
representations to generate the debiased representations, which are used to
produce the final debiased outputs. Experiments across various prompts, PLMs,
and benchmarks show that our approach can not only correct the overfitted
performance caused by prompt bias, but also significantly improve the prompt
retrieval capability (up to 10% absolute performance gain). These results
indicate that our approach effectively alleviates prompt bias in knowledge
evaluation, thereby enhancing the reliability of benchmark assessments.
Hopefully, our plug-and-play approach can be a golden standard to strengthen
PLMs toward reliable knowledge bases. Code and data are released in
https://github.com/FelliYang/PromptBias.
