  Mainstream machine learning conferences have seen a dramatic increase in the
number of participants, along with a growing range of perspectives, in recent
years. Members of the machine learning community are likely to overhear
allegations ranging from randomness of acceptance decisions to institutional
bias. In this work, we critically analyze the review process through a
comprehensive study of papers submitted to ICLR between 2017 and 2020. We
quantify reproducibility/randomness in review scores and acceptance decisions,
and examine whether scores correlate with paper impact. Our findings suggest
strong institutional bias in accept/reject decisions, even after controlling
for paper quality. Furthermore, we find evidence for a gender gap, with female
authors receiving lower scores, lower acceptance rates, and fewer citations per
paper than their male counterparts. We conclude our work with recommendations
for future conference organizers.
