  Journals and conferences worry that peer reviews assisted by artificial
intelligence (AI), in particular, large language models (LLMs), may negatively
influence the validity and fairness of the peer-review system, a cornerstone of
modern science. In this work, we address this concern with a quasi-experimental
study of the prevalence and impact of AI-assisted peer reviews in the context
of the 2024 International Conference on Learning Representations (ICLR), a
large and prestigious machine-learning conference. Our contributions are
threefold. Firstly, we obtain a lower bound for the prevalence of AI-assisted
reviews at ICLR 2024 using the GPTZero LLM detector, estimating that at least
$15.8\%$ of reviews were written with AI assistance. Secondly, we estimate the
impact of AI-assisted reviews on submission scores. Considering pairs of
reviews with different scores assigned to the same paper, we find that in
$53.4\%$ of pairs the AI-assisted review scores higher than the human review
($p = 0.002$; relative difference in probability of scoring higher: $+14.4\%$
in favor of AI-assisted reviews). Thirdly, we assess the impact of receiving an
AI-assisted peer review on submission acceptance. In a matched study,
submissions near the acceptance threshold that received an AI-assisted peer
review were $4.9$ percentage points ($p = 0.024$) more likely to be accepted
than submissions that did not. Overall, we show that AI-assisted reviews are
consequential to the peer-review process and offer a discussion on future
implications of current trends
