  Researchers are increasingly using language models (LMs) for text annotation.
These approaches rely only on a prompt telling the model to return a given
output according to a set of instructions. The reproducibility of LM outputs
may nonetheless be vulnerable to small changes in the prompt design. This calls
into question the replicability of classification routines. To tackle this
problem, researchers have typically tested a variety of semantically similar
prompts to determine what we call "prompt stability." These approaches remain
ad-hoc and task specific. In this article, we propose a general framework for
diagnosing prompt stability by adapting traditional approaches to intra- and
inter-coder reliability scoring. We call the resulting metric the Prompt
Stability Score (PSS) and provide a Python package PromptStability for its
estimation. Using six different datasets and twelve outcomes, we classify >150k
rows of data to: a) diagnose when prompt stability is low; and b) demonstrate
the functionality of the package. We conclude by providing best practice
recommendations for applied researchers.
