  Large language models (LLMs) face issues in handling factual knowledge,
making it vital to evaluate their true ability to understand facts. In this
study, we introduce knowledge probing frameworks, BELIEF(-ICL), to evaluate the
knowledge understanding ability of not only encoder-based PLMs but also
decoder-based PLMs from diverse perspectives. BELIEFs utilize a multi-prompt
dataset to evaluate PLM's accuracy, consistency, and reliability in factual
knowledge understanding. To provide a more reliable evaluation with BELIEFs, we
semi-automatically create MyriadLAMA, which has more diverse prompts than
existing datasets. We validate the effectiveness of BELIEFs in correctly and
comprehensively evaluating PLM's factual understanding ability through
extensive evaluations. We further investigate key factors in learning facts in
LLMs, and reveal the limitation of the prompt-based knowledge probing. The
dataset is anonymously publicized.
