  The recent development of large language models (LLMs) has spurred
discussions about whether LLM-generated "synthetic samples" could complement or
replace traditional surveys, considering their training data potentially
reflects attitudes and behaviors prevalent in the population. A number of
mostly US-based studies have prompted LLMs to mimic survey respondents, with
some of them finding that the responses closely match the survey data. However,
several contextual factors related to the relationship between the respective
target population and LLM training data might affect the generalizability of
such findings. In this study, we investigate the extent to which LLMs can
estimate public opinion in Germany, using the example of vote choice. We
generate a synthetic sample of personas matching the individual characteristics
of the 2017 German Longitudinal Election Study respondents. We ask the LLM
GPT-3.5 to predict each respondent's vote choice and compare these predictions
to the survey-based estimates on the aggregate and subgroup levels. We find
that GPT-3.5 does not predict citizens' vote choice accurately, exhibiting a
bias towards the Green and Left parties. While the LLM captures the tendencies
of "typical" voter subgroups, such as partisans, it misses the multifaceted
factors swaying individual voter choices. By examining the LLM-based prediction
of voting behavior in a new context, our study contributes to the growing body
of research about the conditions under which LLMs can be leveraged for studying
public opinion. The findings point to disparities in opinion representation in
LLMs and underscore the limitations in applying them for public opinion
estimation.
