  The rapid evolution of large language models (LLMs) has ushered in the need
for comprehensive assessments of their performance across various dimensions.
In this paper, we propose LFED, a Literary Fiction Evaluation Dataset, which
aims to evaluate the capability of LLMs on the long fiction comprehension and
reasoning. We collect 95 literary fictions that are either originally written
in Chinese or translated into Chinese, covering a wide range of topics across
several centuries. We define a question taxonomy with 8 question categories to
guide the creation of 1,304 questions. Additionally, we conduct an in-depth
analysis to ascertain how specific attributes of literary fictions (e.g., novel
types, character numbers, the year of publication) impact LLM performance in
evaluations. Through a series of experiments with various state-of-the-art
LLMs, we demonstrate that these models face considerable challenges in
effectively addressing questions related to literary fictions, with ChatGPT
reaching only 57.08% under the zero-shot setting. The dataset will be publicly
available at https://github.com/tjunlp-lab/LFED.git
