  Large language models (LLMs) are remarkable data annotators. They can be used
to generate high-fidelity supervised training data, as well as survey and
experimental data. With the widespread adoption of LLMs, human gold--standard
annotations are key to understanding the capabilities of LLMs and the validity
of their results. However, crowdsourcing, an important, inexpensive way to
obtain human annotations, may itself be impacted by LLMs, as crowd workers have
financial incentives to use LLMs to increase their productivity and income. To
investigate this concern, we conducted a case study on the prevalence of LLM
usage by crowd workers. We reran an abstract summarization task from the
literature on Amazon Mechanical Turk and, through a combination of keystroke
detection and synthetic text classification, estimate that 33-46% of crowd
workers used LLMs when completing the task. Although generalization to other,
less LLM-friendly tasks is unclear, our results call for platforms,
researchers, and crowd workers to find new ways to ensure that human data
remain human, perhaps using the methodology proposed here as a stepping stone.
Code/data: https://github.com/epfl-dlab/GPTurk
