  Crowdsourcing offers a practical method for ranking and scoring large amounts
of items. To investigate the algorithms and incentives that can be used in
crowdsourcing quality evaluations, we built CrowdGrader, a tool that lets
students submit and collaboratively grade solutions to homework assignments. We
present the algorithms and techniques used in CrowdGrader, and we describe our
results and experience in using the tool for several computer-science
assignments.
  CrowdGrader combines the student-provided grades into a consensus grade for
each submission using a novel crowdsourcing algorithm that relies on a
reputation system. The algorithm iterativerly refines inter-dependent estimates
of the consensus grades, and of the grading accuracy of each student. On
synthetic data, the algorithm performs better than alternatives not based on
reputation. On our preliminary experimental data, the performance seems
dependent on the nature of review errors, with errors that can be ascribed to
the reviewer being more tractable than those arising from random external
events. To provide an incentive for reviewers, the grade each student receives
in an assignment is a combination of the consensus grade received by their
submissions, and of a reviewing grade capturing their reviewing effort and
accuracy. This incentive worked well in practice.
