  The ability of CodeLLMs to generate executable and functionally correct code
at the repository-level scale remains largely unexplored. We introduce
RepoExec, a novel benchmark for evaluating code generation at the
repository-level scale. RepoExec focuses on three main aspects: executability,
functional correctness through automated test case generation with high
coverage rate, and carefully crafted cross-file contexts to accurately generate
code. Our work explores a controlled scenario where developers specify
necessary code dependencies, challenging the model to integrate these
accurately. Experiments show that while pretrained LLMs outperform
instruction-tuned models in correctness, the latter excel in utilizing provided
dependencies and demonstrating debugging capabilities. We also introduce a new
instruction-tuned dataset that focuses on code dependencies and demonstrate
that CodeLLMs fine-tuned on our dataset have a better capability to leverage
these dependencies effectively. RepoExec aims to provide a comprehensive
evaluation of code functionality and alignment with developer intent, paving
the way for more reliable and applicable CodeLLMs in real-world scenarios. The
dataset and source code can be found
at~\url{https://github.com/FSoft-AI4Code/RepoExec}.
