  Program synthesis has been long studied with recent approaches focused on
directly using the power of Large Language Models (LLMs) to generate code.
Programming benchmarks, with curated synthesis problems and test-cases, are
used to measure the performance of various LLMs on code synthesis. However,
these test-cases can be limited in both quantity and quality for fully
assessing the functional correctness of the generated code. Such limitation in
the existing benchmarks begs the following question: In the era of LLMs, is the
code generated really correct? To answer this, we propose EvalPlus -- a code
synthesis evaluation framework to rigorously benchmark the functional
correctness of LLM-synthesized code. EvalPlus augments a given evaluation
dataset with large amounts of test-cases newly produced by an automatic test
input generator, powered by both LLM- and mutation-based strategies. While
EvalPlus is general, we extend the test-cases of the popular HumanEval
benchmark by 80x to build HumanEval+. Our extensive evaluation across 26
popular LLMs (e.g., GPT-4 and ChatGPT) demonstrates that HumanEval+ is able to
catch significant amounts of previously undetected wrong code synthesized by
LLMs, reducing the pass@k by up-to 19.3-28.9%. We also surprisingly found that
test insufficiency can lead to mis-ranking. For example, both
WizardCoder-CodeLlama and Phind-CodeLlama now outperform ChatGPT on HumanEval+,
while none of them could on HumanEval. Our work not only indicates that prior
popular code synthesis evaluation results do not accurately reflect the true
performance of LLMs for code synthesis, but also opens up a new direction to
improve such programming benchmarks through automated testing. We have
open-sourced our tools, enhanced datasets as well as all LLM-generated code at
https://github.com/evalplus/evalplus to facilitate and accelerate future
LLM-for-code research.
