  When asked, large language models (LLMs) like ChatGPT claim that they can
assist with relevance judgments but it is not clear whether automated judgments
can reliably be used in evaluations of retrieval systems. In this perspectives
paper, we discuss possible ways for LLMs to support relevance judgments along
with concerns and issues that arise. We devise a human--machine collaboration
spectrum that allows to categorize different relevance judgment strategies,
based on how much humans rely on machines. For the extreme point of "fully
automated judgments", we further include a pilot experiment on whether
LLM-based relevance judgments correlate with judgments from trained human
assessors. We conclude the paper by providing opposing perspectives for and
against the use of~LLMs for automatic relevance judgments, and a compromise
perspective, informed by our analyses of the literature, our preliminary
experimental evidence, and our experience as IR researchers.
