  In recent years, substantial advancements have been made in the development
of large language models, achieving remarkable performance across diverse
tasks. To evaluate the knowledge ability of language models, previous studies
have proposed lots of benchmarks based on question-answering pairs. We argue
that it is not reliable and comprehensive to evaluate language models with a
fixed question or limited paraphrases as the query, since language models are
sensitive to prompt. Therefore, we introduce a novel concept named knowledge
boundary to encompass both prompt-agnostic and prompt-sensitive knowledge
within language models. Knowledge boundary avoids prompt sensitivity in
language model evaluations, rendering them more dependable and robust. To
explore the knowledge boundary for a given model, we propose projected gradient
descent method with semantic constraints, a new algorithm designed to identify
the optimal prompt for each piece of knowledge. Experiments demonstrate a
superior performance of our algorithm in computing the knowledge boundary
compared to existing methods. Furthermore, we evaluate the ability of multiple
language models in several domains with knowledge boundary.
