  The task of generating code solutions for a given programming problem can
benefit from the use of pre-trained language models such as Codex, which can
produce multiple diverse samples. However, a major challenge for this task is
to select the most appropriate solution from the multiple samples generated by
the pre-trained language models. A natural way to evaluate the quality and
correctness of a code solution is to run it against a set of test cases, but
the manual creation of such test cases is often costly and time-consuming. In
this paper, we propose a novel method, CodeT, that leverages the same
pre-trained language models to automatically generate test cases for the code
samples, thus reducing the human effort and increasing the coverage of the test
scenarios. CodeT then executes the code samples using the generated test cases,
and performs a dual execution agreement, which considers both the consistency
of the outputs against the generated test cases and the agreement of the
outputs with other code samples. We conduct comprehensive experiments on four
benchmarks, HumanEval, MBPP, APPS and CodeContests, using five different
pre-trained language models with varying sizes and capabilities. Our results
show that CodeT can significantly improve the performance of code solution
selection over previous methods, achieving remarkable and consistent gains
across different models and benchmarks. For instance, CodeT improves the pass@1
metric on HumanEval to 65.8%, which represents an absolute improvement of 18.8%
over the code-davinci-002 model, and an absolute improvement of more than 20%
over the previous state-of-the-art results.
