  The increasing volume of scientific publications and grant proposals has
generated an unprecedentedly high workload to scientific communities.
Consequently, review quality has been decreasing and review outcomes have
become less correlated with the real merits of the papers and proposals. A
novel distributed peer review (DPR) approach has recently been proposed to
address these issues. The new approach assigns principal investigators (PIs)
who submitted proposals (or papers) to the same program as reviewers. Each PI
reviews and ranks a small number (such as seven) of other PIs' proposals. The
individual rankings are then used to estimate a global ranking of all proposals
using the Modified Borda Count (MBC). In this study, we perform simulation
studies to investigate several parameters important for the decision making
when adopting this new approach. We also propose a new method called
Concordance Index-based Global Ranking (CIGR) to estimate global ranking from
individual rankings. An efficient simulated annealing algorithm is designed to
search the optimal Concordance Index (CI). Moreover, we design a new balanced
review assignment procedure, which can result in significantly better
performance for both MBC and CIGR methods. We found that CIGR performs better
than MBC when the review quality is relatively high. As review quality and
review difficulty are tightly correlated, we constructed a boundary in the
space of review quality vs review difficulty that separates the CIGR-superior
and MBC-superior regions. Finally, we propose a multi-stage DPR strategy based
on CIGR, which has the potential to substantially improve the overall review
performance while reducing the review workload.
