  ChatGPT and other large language models (LLMs) have proven useful in
crowdsourcing tasks, where they can effectively annotate machine learning
training data. However, this means that they also have the potential for
misuse, specifically to automatically answer surveys. LLMs can potentially
circumvent quality assurance measures, thereby threatening the integrity of
methodologies that rely on crowdsourcing surveys. In this paper, we propose a
mechanism to detect LLM-generated responses to surveys. The mechanism uses
"prompt injection", such as directions that can mislead LLMs into giving
predictable responses. We evaluate our technique against a range of question
scenarios, types, and positions, and find that it can reliably detect
LLM-generated responses with more than 93% effectiveness. We also provide an
open-source software to help survey designers use our technique to detect LLM
responses. Our work is a step in ensuring that survey methodologies remain
rigorous vis-a-vis LLMs.
