  In light of recent breakthroughs in large language models (LLMs) that have
revolutionized natural language processing (NLP), there is an urgent need for
new benchmarks to keep pace with the fast development of LLMs. In this paper,
we propose CFLUE, the Chinese Financial Language Understanding Evaluation
benchmark, designed to assess the capability of LLMs across various dimensions.
Specifically, CFLUE provides datasets tailored for both knowledge assessment
and application assessment. In knowledge assessment, it consists of 38K+
multiple-choice questions with associated solution explanations. These
questions serve dual purposes: answer prediction and question reasoning. In
application assessment, CFLUE features 16K+ test instances across distinct
groups of NLP tasks such as text classification, machine translation, relation
extraction, reading comprehension, and text generation. Upon CFLUE, we conduct
a thorough evaluation of representative LLMs. The results reveal that only
GPT-4 and GPT-4-turbo achieve an accuracy exceeding 60\% in answer prediction
for knowledge assessment, suggesting that there is still substantial room for
improvement in current LLMs. In application assessment, although GPT-4 and
GPT-4-turbo are the top two performers, their considerable advantage over
lightweight LLMs is noticeably diminished. The datasets and scripts associated
with CFLUE are openly accessible at https://github.com/aliyun/cflue.
