  Several recent papers have investigated the potential of language models as
knowledge bases as well as the existence of severe biases when extracting
factual knowledge. In this work, we focus on the factual probing performance
over unseen prompts from tuning, and using a probabilistic view we show the
inherent misalignment between pre-training and downstream tuning objectives in
language models for probing knowledge. We hypothesize that simultaneously
debiasing these objectives can be the key to generalisation over unseen
prompts. We propose an adapter-based framework, UniArk, for generalised and
consistent factual knowledge extraction through simple methods without
introducing extra parameters. Extensive experiments show that UniArk can
significantly improve the model's out-of-domain generalisation as well as
consistency under various prompts. Additionally, we construct ParaTrex, a
large-scale and diverse dataset for measuring the inconsistency and
out-of-domain generation of models. Further, ParaTrex offers a reference method
for constructing paraphrased datasets using large language models.
