  Political polling is a multi-billion dollar industry with outsized influence
on the societal trajectory of the United States and nations around the world.
However, it has been challenged by factors that stress its cost, availability,
and accuracy. At the same time, artificial intelligence (AI) chatbots have
become compelling stand-ins for human behavior, powered by increasingly
sophisticated large language models (LLMs). Could AI chatbots be an effective
tool for anticipating public opinion on controversial issues to the extent that
they could be used by campaigns, interest groups, and polling firms? We have
developed a prompt engineering methodology for eliciting human-like survey
responses from ChatGPT, which simulate the response to a policy question of a
person described by a set of demographic factors, and produce both an ordinal
numeric response score and a textual justification. We execute large scale
experiments, querying for thousands of simulated responses at a cost far lower
than human surveys. We compare simulated data to human issue polling data from
the Cooperative Election Study (CES). We find that ChatGPT is effective at
anticipating both the mean level and distribution of public opinion on a
variety of policy issues such as abortion bans and approval of the US Supreme
Court, particularly in their ideological breakdown (correlation typically
>85%). However, it is less successful at anticipating demographic-level
differences. Moreover, ChatGPT tends to overgeneralize to new policy issues
that arose after its training data was collected, such as US support for
involvement in the war in Ukraine. Our work has implications for our
understanding of the strengths and limitations of the current generation of AI
chatbots as virtual publics or online listening platforms, future directions
for LLM development, and applications of AI tools to the political domain.
(Abridged)
