  The use of automatic grading tools has become nearly ubiquitous in large
undergraduate programming courses, and recent work has focused on improving the
quality of automatically generated feedback. However, there is a relative lack
of data directly comparing student outcomes when receiving computer-generated
feedback and human-written feedback. This paper addresses this gap by splitting
one 90-student class into two feedback groups and analyzing differences in the
two cohorts' performance. The class is an intro to AI with programming HW
assignments. One group of students received detailed computer-generated
feedback on their programming assignments describing which parts of the
algorithms' logic was missing; the other group additionally received
human-written feedback describing how their programs' syntax relates to issues
with their logic, and qualitative (style) recommendations for improving their
code. Results on quizzes and exam questions suggest that human feedback helps
students obtain a better conceptual understanding, but analyses found no
difference between the groups' ability to collaborate on the final project. The
course grade distribution revealed that students who received human-written
feedback performed better overall; this effect was the most pronounced in the
middle two quartiles of each group. These results suggest that feedback about
the syntax-logic relation may be a primary mechanism by which human feedback
improves student outcomes.
