  It is common to see a handful of reviewers reject a highly novel paper,
because they view, say, extensive experiments as far more important than
novelty, whereas the community as a whole would have embraced the paper. More
generally, the disparate mapping of criteria scores to final recommendations by
different reviewers is a major source of inconsistency in peer review. In this
paper we present a framework inspired by empirical risk minimization (ERM) for
learning the community's aggregate mapping. The key challenge that arises is
the specification of a loss function for ERM. We consider the class of $L(p,q)$
loss functions, which is a matrix-extension of the standard class of $L_p$
losses on vectors; here the choice of the loss function amounts to choosing the
hyperparameters $p, q \in [1,\infty]$. To deal with the absence of ground truth
in our problem, we instead draw on computational social choice to identify
desirable values of the hyperparameters $p$ and $q$. Specifically, we
characterize $p=q=1$ as the only choice of these hyperparameters that satisfies
three natural axiomatic properties. Finally, we implement and apply our
approach to reviews from IJCAI 2017.
