  Many scientific conferences employ a two-phase paper review process, where
some papers are assigned additional reviewers after the initial reviews are
submitted. Many conferences also design and run experiments on their paper
review process, where some papers are assigned reviewers who provide reviews
under an experimental condition. In this paper, we consider the question: how
should reviewers be divided between phases or conditions in order to maximize
total assignment similarity? We make several contributions towards answering
this question. First, we prove that when the set of papers requiring additional
review is unknown, a simplified variant of this problem is NP-hard. Second, we
empirically show that across several datasets pertaining to real conference
data, dividing reviewers between phases/conditions uniformly at random allows
an assignment that is nearly as good as the oracle optimal assignment. This
uniformly random choice is practical for both the two-phase and conference
experiment design settings. Third, we provide explanations of this phenomenon
by providing theoretical bounds on the suboptimality of this random strategy
under certain natural conditions. From these easily-interpretable conditions,
we provide actionable insights to conference program chairs about whether a
random reviewer split is suitable for their conference.
