  The emergence of large language models (LLMs) has sparked enormous interest
due to their potential application across a range of educational tasks. For
example, recent work in programming education has used LLMs to generate
learning resources, improve error messages, and provide feedback on code.
However, one factor that limits progress within the field is that much of the
research uses bespoke datasets and different evaluation metrics, making direct
comparisons between results unreliable. Thus, there is a pressing need for
standardization and benchmarks that facilitate the equitable comparison of
competing approaches. One task where LLMs show great promise is program repair,
which can be used to provide debugging support and next-step hints to students.
In this article, we propose a novel educational program repair benchmark. We
curate two high-quality publicly available programming datasets, present a
unified evaluation procedure introducing a novel evaluation metric rouge@k for
approximating the quality of repairs, and evaluate a set of five recent models
to establish baseline performance.
