  In courses that involve programming assignments, giving meaningful feedback
to students is an important challenge. Human beings can give useful feedback by
manually grading the programs but this is a time-consuming, labor intensive,
and usually boring process. Automatic graders can be fast and scale well but
they usually provide poor feedback. Although there has been research on
improving automatic graders, research on scaling and improving human grading is
limited.
  We propose to scale human grading by augmenting the manual grading process
with an equivalence algorithm that can identify the equivalences between
student submissions. This enables human graders to give targeted feedback for
multiple student submissions at once. Our technique is conservative in two
aspects. First, it identifies equivalence between submissions that are
algorithmically similar, e.g., it cannot identify the equivalence between
quicksort and mergesort. Second, it uses formal methods instead of clustering
algorithms from the machine learning literature. This allows us to prove a
soundness result that guarantees that submissions will never be clustered
together in error. Despite only reporting equivalence when there is algorithmic
similarity and the ability to formally prove equivalence, we show that our
technique can significantly reduce grading time for thousands of programming
submissions from an introductory functional programming course.
