  Large Language Models (LLMs) demonstrate increasingly human-like abilities
across a wide variety of tasks. In this paper, we investigate whether LLMs like
ChatGPT can accurately infer the psychological dispositions of social media
users and whether their ability to do so varies across socio-demographic
groups. Specifically, we test whether GPT-3.5 and GPT-4 can derive the Big Five
personality traits from users' Facebook status updates in a zero-shot learning
scenario. Our results show an average correlation of r = .29 (range = [.22,
.33]) between LLM-inferred and self-reported trait scores - a level of accuracy
that is similar to that of supervised machine learning models specifically
trained to infer personality. Our findings also highlight heterogeneity in the
accuracy of personality inferences across different age groups and gender
categories: predictions were found to be more accurate for women and younger
individuals on several traits, suggesting a potential bias stemming from the
underlying training data or differences in online self-expression. The ability
of LLMs to infer psychological dispositions from user-generated text has the
potential to democratize access to cheap and scalable psychometric assessments
for both researchers and practitioners. On the one hand, this democratization
might facilitate large-scale research of high ecological validity and spark
innovation in personalized services. On the other hand, it also raises ethical
concerns regarding user privacy and self-determination, highlighting the need
for stringent ethical frameworks and regulation.
