  Large Language Models (LLMs) are capable of successfully performing many
language processing tasks zero-shot (without training data). If zero-shot LLMs
can also reliably classify and explain social phenomena like persuasiveness and
political ideology, then LLMs could augment the Computational Social Science
(CSS) pipeline in important ways. This work provides a road map for using LLMs
as CSS tools. Towards this end, we contribute a set of prompting best practices
and an extensive evaluation pipeline to measure the zero-shot performance of 13
language models on 25 representative English CSS benchmarks. On taxonomic
labeling tasks (classification), LLMs fail to outperform the best fine-tuned
models but still achieve fair levels of agreement with humans. On free-form
coding tasks (generation), LLMs produce explanations that often exceed the
quality of crowdworkers' gold references. We conclude that the performance of
today's LLMs can augment the CSS research pipeline in two ways: (1) serving as
zero-shot data annotators on human annotation teams, and (2) bootstrapping
challenging creative generation tasks (e.g., explaining the underlying
attributes of a text). In summary, LLMs are posed to meaningfully participate
in social science analysis in partnership with humans.
