  Previous works on Large Language Models (LLMs) have mainly focused on
evaluating their helpfulness or harmlessness. However, honesty, another crucial
alignment criterion, has received relatively less attention. Dishonest
behaviors in LLMs, such as spreading misinformation and defrauding users,
present severe risks that intensify as these models approach superintelligent
levels. Enhancing honesty in LLMs addresses critical limitations and helps
uncover latent capabilities that are not readily expressed. This underscores
the urgent need for reliable methods and benchmarks to effectively ensure and
evaluate the honesty of LLMs.
  In this paper, we introduce BeHonest, a pioneering benchmark specifically
designed to assess honesty in LLMs comprehensively. BeHonest evaluates three
essential aspects of honesty: awareness of knowledge boundaries, avoidance of
deceit, and consistency in responses. Building on this foundation, we designed
10 scenarios to evaluate and analyze 9 popular LLMs on the market, including
both closed-source and open-source models from different model families with
varied model sizes. Our findings indicate that there is still significant room
for improvement in the honesty of LLMs. We encourage the AI community to
prioritize honesty alignment in these models, which can harness their full
potential to benefit society while preventing them from causing harm through
deception or inconsistency. Our benchmark and code can be found at:
\url{https://github.com/GAIR-NLP/BeHonest}.
