  Quality assurance remains a key topic in human computation research. Prior
work indicates that majority voting is effective for low difficulty tasks, but
has limitations for harder tasks. This paper explores two methods of addressing
this problem: tournament selection and elimination selection, which exploit 2-,
3- and 4-way comparisons between different answers to human computation tasks.
Our experimental results and statistical analyses show that both methods
produce the correct answer in noisy human computation environment more often
than majority voting. Furthermore, we find that the use of 4-way comparisons
can significantly reduce the cost of quality assurance relative to the use of
2-way comparisons.
