  With the growing reliance on automated code completion tools in software
development, the need for robust evaluation benchmarks has become critical.
However, existing benchmarks focus more on code generation tasks in function
and class level and provide rich text description to prompt the model. By
contrast, such descriptive prompt is commonly unavailable in real development
and code completion can occur in wider range of situations such as in the
middle of a function or a code block. These limitations makes the evaluation
poorly align with the practical scenarios of code completion tools. In this
paper, we propose RepoMasterEval, a novel benchmark for evaluating code
completion models constructed from real-world Python and TypeScript
repositories. Each benchmark datum is generated by masking a code snippet
(ground truth) from one source code file with existing test suites. To improve
test accuracy of model generated code, we employ mutation testing to measure
the effectiveness of the test cases and we manually crafted new test cases for
those test suites with low mutation score. Our empirical evaluation on 6
state-of-the-art models shows that test argumentation is critical in improving
the accuracy of the benchmark and RepoMasterEval is able to report difference
in model performance in real-world scenarios. The deployment of RepoMasterEval
in a collaborated company for one month also revealed that the benchmark is
useful to give accurate feedback during model training and the score is in high
correlation with the model's performance in practice. Based on our findings, we
call for the software engineering community to build more LLM benchmarks
tailored for code generation tools taking the practical and complex development
environment into consideration.
