  Program repair techniques offer cost-saving benefits for debugging within
software development and programming education scenarios. With the proven
effectiveness of Large Language Models (LLMs) in code-related tasks,
researchers have explored their potential for program repair. However, it is
crucial to recognize that existing repair benchmarks may have influenced LLM
training data, potentially causing data leakage. To evaluate LLMs' realistic
repair capabilities, (1) we introduce an extensive, non-crawled benchmark,
referred to as TutorCode, comprising 1,239 C++ defect codes and associated
information such as tutor guidance, solution description, failing test cases,
and the corrected code. Our work assesses the repair performance of 12 LLMs on
TutorCode, measuring repair correctness (TOP-5 and AVG-5) and patch precision
(RPSR). (2) We then provide a comprehensive investigation into which types of
extra information can help LLMs improve their performance in repairing defects.
Among these types, tutor guidance was found to be the most effective
information in enhancing LLM repair capabilities. To fully harness LLMs'
conversational capabilities and the benefits of augmented information, (3) we
introduce a novel conversational semi-automatic repair framework CREF assisting
human tutor. It demonstrates a remarkable AVG-5 improvement of 17.2%-24.6%
compared to the baseline, achieving an impressive AVG-5 of 76.6% when utilizing
GPT-4. These results highlight the potential for enhancing LLMs' repair
capabilities through interactions with tutors and historical conversations
involving incorrect responses. The successful application of CREF in a
real-world educational setting demonstrates its effectiveness in reducing
tutors' workload and improving students' learning experience, while also
showcasing its promise for facilitating other software engineering tasks, such
as code review.
