  Very large language models (LLMs) perform extremely well on a spectrum of NLP
tasks in a zero-shot setting. However, little is known about their performance
on human-level NLP problems which rely on understanding psychological concepts,
such as assessing personality traits. In this work, we investigate the
zero-shot ability of GPT-3 to estimate the Big 5 personality traits from users'
social media posts. Through a set of systematic experiments, we find that
zero-shot GPT-3 performance is somewhat close to an existing pre-trained SotA
for broad classification upon injecting knowledge about the trait in the
prompts. However, when prompted to provide fine-grained classification, its
performance drops to close to a simple most frequent class (MFC) baseline. We
further analyze where GPT-3 performs better, as well as worse, than a
pretrained lexical model, illustrating systematic errors that suggest ways to
improve LLMs on human-level NLP tasks.
