  Large language models such as Codex, have shown the capability to produce
code for many programming tasks. However, the success rate of existing models
is low, especially for complex programming tasks. One of the reasons is that
language models lack awareness of program semantics, resulting in incorrect
programs, or even programs which do not compile. In this paper, we
systematically study whether automated program repair (APR) techniques can fix
the incorrect solutions produced by language models in LeetCode contests. The
goal is to study whether APR techniques can enhance reliability in the code
produced by large language models. Our study revealed that: (1) automatically
generated code shares common programming mistakes with human-crafted solutions,
indicating APR techniques may have potential to fix auto-generated code; (2)
given bug location information provided by a statistical fault localization
approach, the newly released Codex edit mode, which supports editing code, is
similar to or better than existing Java repair tools TBar and Recoder in fixing
incorrect solutions. By analyzing the experimental results generated by these
tools, we provide several suggestions: (1) enhancing APR tools to surpass
limitations in patch space (e.g., introducing more flexible fault localization)
is desirable; (2) as large language models can derive more fix patterns by
training on more data, future APR tools could shift focus from adding more fix
patterns to synthesis/semantics based approaches, (3) combination of language
models with APR to curate patch ingredients, is worth studying.
