  Large language models (LLMs) have demonstrated impressive reasoning
capabilities, yet there is ongoing debate about these abilities and the
potential data contamination problem recently. This paper aims to evaluate the
reasoning capacities of LLMs, specifically in solving recent competition-level
programming problems in Codeforces, which are expert-crafted and unique,
requiring deep understanding and robust reasoning skills. We first provide a
comprehensive evaluation of GPT-4's peiceived zero-shot performance on this
task, considering various aspects such as problems' release time, difficulties,
and types of errors encountered. Surprisingly, the peiceived performance of
GPT-4 has experienced a cliff like decline in problems after September 2021
consistently across all the difficulties and types of problems, which shows the
potential data contamination, as well as the challenges for any existing LLM to
solve unseen complex reasoning problems. We further explore various approaches
such as fine-tuning, Chain-of-Thought prompting and problem description
simplification, unfortunately none of them is able to consistently mitigate the
challenges. Through our work, we emphasis the importance of this excellent data
source for assessing the genuine reasoning capabilities of LLMs, and foster the
development of LLMs with stronger reasoning abilities and better generalization
in the future.
