  The LLM Agent, equipped with a code interpreter, is capable of automatically
solving real-world coding tasks, such as data analysis and image editing.
  However, existing benchmarks primarily focus on either simplistic tasks, such
as completing a few lines of code, or on extremely complex and specific tasks
at the repository level, neither of which are representative of various daily
coding tasks.
  To address this gap, we introduce \textbf{PyBench}, a benchmark encompassing
five main categories of real-world tasks, covering more than 10 types of files.
Given a high-level user query and related files, the LLM Agent needs to reason
and execute Python code via a code interpreter for a few turns before making a
formal response to fulfill the user's requirements. Successfully addressing
tasks in PyBench demands a robust understanding of various Python packages,
superior reasoning capabilities, and the ability to incorporate feedback from
executed code. Our evaluations indicate that current open-source LLMs are
struggling with these tasks. Hence, we conduct analysis and experiments on four
kinds of datasets proving that comprehensive abilities are needed for PyBench.
Our fine-tuned 8B size model: \textbf{PyLlama3} achieves an exciting
performance on PyBench which surpasses many 33B and 70B size models. Our
Benchmark, Training Dataset, and Model are available at:
{https://github.com/Mercury7353/PyBench}
