  Large language models (LLMs) have demonstrated exceptional performance in
various natural language processing tasks, yet their efficacy in more
challenging and domain-specific tasks remains largely unexplored. This paper
presents FinEval, a benchmark specifically designed for the financial domain
knowledge in the LLMs. FinEval is a collection of high-quality multiple-choice
questions covering Finance, Economy, Accounting, and Certificate. It includes
4,661 questions spanning 34 different academic subjects. To ensure a
comprehensive model performance evaluation, FinEval employs a range of prompt
types, including zero-shot and few-shot prompts, as well as answer-only and
chain-of-thought prompts. Evaluating state-of-the-art Chinese and English LLMs
on FinEval, the results show that only GPT-4 achieved an accuracy close to 70%
in different prompt settings, indicating significant growth potential for LLMs
in the financial domain knowledge. Our work offers a more comprehensive
financial knowledge evaluation benchmark, utilizing data of mock exams and
covering a wide range of evaluated LLMs.
