  We introduce Ghostbuster, a state-of-the-art system for detecting
AI-generated text. Our method works by passing documents through a series of
weaker language models, running a structured search over possible combinations
of their features, and then training a classifier on the selected features to
predict whether documents are AI-generated. Crucially, Ghostbuster does not
require access to token probabilities from the target model, making it useful
for detecting text generated by black-box models or unknown model versions. In
conjunction with our model, we release three new datasets of human- and
AI-generated text as detection benchmarks in the domains of student essays,
creative writing, and news articles. We compare Ghostbuster to a variety of
existing detectors, including DetectGPT and GPTZero, as well as a new RoBERTa
baseline. Ghostbuster achieves 99.0 F1 when evaluated across domains, which is
5.9 F1 higher than the best preexisting model. It also outperforms all previous
approaches in generalization across writing domains (+7.5 F1), prompting
strategies (+2.1 F1), and language models (+4.4 F1). We also analyze the
robustness of our system to a variety of perturbations and paraphrasing attacks
and evaluate its performance on documents written by non-native English
speakers.
