  Automatic program repair papers tend to repeatedly use the same benchmarks.
This poses a threat to the external validity of the findings of the program
repair research community. In this paper, we perform an empirical study of
automatic repair on a benchmark of bugs called QuixBugs, which has been little
studied. In this paper, 1) We report on the characteristics of QuixBugs; 2) We
study the effectiveness of 10 program repair tools on it; 3) We apply three
patch correctness assessment techniques to comprehensively study the presence
of overfitting patches in QuixBugs. Our key results are: 1) 16/40 buggy
programs in QuixBugs can be repaired with at least a test suite adequate patch;
2) A total of 338 plausible patches are generated on the QuixBugs by the
considered tools, and 53.3% of them are overfitting patches according to our
manual assessment; 3) The three automated patch correctness assessment
techniques, RGT_Evosuite, RGT_InputSampling and GT_Invariants, achieve an
accuracy of 98.2%, 80.8% and 58.3% in overfitting detection, respectively. To
our knowledge, this is the largest empirical study of automatic repair on
QuixBugs, combining both quantitative and qualitative insights. All our
empirical results are publicly available on GitHub in order to facilitate
future research on automatic program repair.
