  A major threat to the peer-review systems of computer science conferences is
the existence of "collusion rings" between reviewers. In such collusion rings,
reviewers who have also submitted their own papers to the conference work
together to manipulate the conference's paper assignment, with the aim of being
assigned to review each other's papers. The most straightforward way that
colluding reviewers can manipulate the paper assignment is by indicating their
interest in each other's papers through strategic paper bidding. One potential
approach to solve this important problem would be to detect the colluding
reviewers from their manipulated bids, after which the conference can take
appropriate action. While prior work has developed effective techniques to
detect other kinds of fraud, no research has yet established that detecting
collusion rings is even possible. In this work, we tackle the question of
whether it is feasible to detect collusion rings from the paper bidding. To
answer this question, we conduct empirical analysis of two realistic conference
bidding datasets, including evaluations of existing algorithms for fraud
detection in other applications. We find that collusion rings can achieve
considerable success at manipulating the paper assignment while remaining
hidden from detection: for example, in one dataset, undetected colluders are
able to achieve assignment to up to 30% of the papers authored by other
colluders. In addition, when 10 colluders bid on all of each other's papers, no
detection algorithm outputs a group of reviewers with more than 31% overlap
with the true colluders. These results suggest that collusion cannot be
effectively detected from the bidding using popular existing tools,
demonstrating the need to develop more complex detection algorithms as well as
those that leverage additional metadata (e.g., reviewer-paper text-similarity
scores).
