  In peer review, reviewers are usually asked to provide scores for the papers.
The scores are then used by Area Chairs or Program Chairs in various ways in
the decision-making process. The scores are usually elicited in a quantized
form to accommodate the limited cognitive ability of humans to describe their
opinions in numerical values. It has been found that the quantized scores
suffer from a large number of ties, thereby leading to a significant loss of
information. To mitigate this issue, conferences have started to ask reviewers
to additionally provide a ranking of the papers they have reviewed. There are
however two key challenges. First, there is no standard procedure for using
this ranking information and Area Chairs may use it in different ways
(including simply ignoring them), thereby leading to arbitrariness in the
peer-review process. Second, there are no suitable interfaces for judicious use
of this data nor methods to incorporate it in existing workflows, thereby
leading to inefficiencies. We take a principled approach to integrate the
ranking information into the scores. The output of our method is an updated
score pertaining to each review that also incorporates the rankings. Our
approach addresses the two aforementioned challenges by: (i) ensuring that
rankings are incorporated into the updates scores in the same manner for all
papers, thereby mitigating arbitrariness, and (ii) allowing to seamlessly use
existing interfaces and workflows designed for scores. We empirically evaluate
our method on synthetic datasets as well as on peer reviews from the ICLR 2017
conference, and find that it reduces the error by approximately 30% as compared
to the best performing baseline on the ICLR 2017 data.
