  On Wikipedia, sophisticated algorithmic tools are used to assess the quality
of edits and take corrective actions. However, algorithms can fail to solve the
problems they were designed for if they conflict with the values of communities
who use them. In this study, we take a Value-Sensitive Algorithm Design
approach to understanding a community-created and -maintained machine
learning-based algorithm called the Objective Revision Evaluation System
(ORES)---a quality prediction system used in numerous Wikipedia applications
and contexts. Five major values converged across stakeholder groups that ORES
(and its dependent applications) should: (1) reduce the effort of community
maintenance, (2) maintain human judgement as the final authority, (3) support
differing peoples' differing workflows, (4) encourage positive engagement with
diverse editor groups, and (5) establish trustworthiness of people and
algorithms within the community. We reveal tensions between these values and
discuss implications for future research to improve algorithms like ORES.
