  Recently, Large Language Models (LLMs) make remarkable evolutions in language
understanding and generation. Following this, various benchmarks for measuring
all kinds of capabilities of LLMs have sprung up. In this paper, we challenge
the reasoning and understanding abilities of LLMs by proposing a FaLlacy
Understanding Benchmark (FLUB) containing cunning texts that are easy for
humans to understand but difficult for models to grasp. Specifically, the
cunning texts that FLUB focuses on mainly consist of the tricky, humorous, and
misleading texts collected from the real internet environment. And we design
three tasks with increasing difficulty in the FLUB benchmark to evaluate the
fallacy understanding ability of LLMs. Based on FLUB, we investigate the
performance of multiple representative and advanced LLMs, reflecting our FLUB
is challenging and worthy of more future study. Interesting discoveries and
valuable insights are achieved in our extensive experiments and detailed
analyses. We hope that our benchmark can encourage the community to improve
LLMs' ability to understand fallacies. Our data and codes are available at
https://github.com/THUKElab/FLUB.
