  It is common to evaluate a set of items by soliciting people to rate them.
For example, universities ask students to rate the teaching quality of their
instructors, and conference organizers ask authors of submissions to evaluate
the quality of the reviews. However, in these applications, students often give
a higher rating to a course if they receive higher grades in a course, and
authors often give a higher rating to the reviews if their papers are accepted
to the conference. In this work, we call these external factors the "outcome"
experienced by people, and consider the problem of mitigating these
outcome-induced biases in the given ratings when some information about the
outcome is available. We formulate the information about the outcome as a known
partial ordering on the bias. We propose a debiasing method by solving a
regularized optimization problem under this ordering constraint, and also
provide a carefully designed cross-validation method that adaptively chooses
the appropriate amount of regularization. We provide theoretical guarantees on
the performance of our algorithm, as well as experimental evaluations.
