  Automated program repair (APR) aims to help developers improve software
reliability by generating patches for buggy programs. Although many code
language models (CLM) are developed and effective in many software tasks such
as code completion, there has been little comprehensive, in-depth work to
evaluate CLMs' fixing capabilities and to fine-tune CLMs for the APR task.
  Firstly, this work is the first to evaluate ten CLMs on four APR benchmarks,
which shows that surprisingly, the best CLM, as is, fixes 72% more bugs than
the state-of-the-art deep-learning (DL)-based APR techniques. Secondly, one of
the four APR benchmarks was created by us in this paper to avoid data leaking
for a fair evaluation. Thirdly, it is the first work to fine-tune CLMs with APR
training data, which shows that fine-tuning brings 31%-1,267% improvement to
CLMs and enables them to fix 46%-164% more bugs than existing DL-based APR
techniques. Fourthly, this work studies the impact of buggy lines, showing that
CLMs, as is, cannot make good use of the buggy lines to fix bugs, yet
fine-tuned CLMs could potentially over-rely on buggy lines. Lastly, this work
analyzes the size, time, and memory efficiency of different CLMs.
  This work shows promising directions for the APR domain, such as fine-tuning
CLMs with APR-specific designs, and also raises awareness of fair and
comprehensive evaluations of CLMs and calls for more transparent reporting of
open-source repositories used in the pre-training data to address the data
leaking problem.
