  Large Language Models (LLMs) have shown promise in automated code generation
but typically excel only in simpler tasks such as generating standalone code
units. Real-world software development, however, often involves complex code
repositories (named repo) with complex dependencies and extensive
documentation. To fill this gap, our research pivots towards evaluating LLMs in
a more realistic setting -- real-world repo-level code generation. We introduce
CodeAgentBench, a manually curated benchmark for repo-level code generation.
This benchmark comprises five high-quality Python projects, encompassing a
total of 101 samples. We assess nine leading LLMs on repo-level tasks and
observe a decline in their performance. To tackle this, we present CodeAgent, a
novel LLM-based agent framework that employs external tools for effective
repo-level code generation. CodeAgent integrates five programming tools,
enabling interaction with software artifacts for information retrieval, code
symbol navigation, and code testing. We implement four agent strategies to
optimize these tools' usage. Our experiments on CodeAgentBench show that
CodeAgent enhances LLM performance significantly, with improvements ranging
from 18.1\% to 250\%. Further tests on the HumanEval benchmark confirm
CodeAgent's adaptability and efficacy across various code generation tasks.
Notably, CodeAgent outperforms commercial products like Github Copilot,
showcasing superior accuracy and efficiency. These results demonstrate
CodeAgent's robust capabilities in code generation, highlighting its potential
for real-world repo-level coding challenges.
