  Despite decades of research on authorship attribution (AA) and authorship
verification (AV), inconsistent dataset splits/filtering and mismatched
evaluation methods make it difficult to assess the state of the art. In this
paper, we present a survey of the fields, resolve points of confusion,
introduce Valla that standardizes and benchmarks AA/AV datasets and metrics,
provide a large-scale empirical evaluation, and provide apples-to-apples
comparisons between existing methods. We evaluate eight promising methods on
fifteen datasets (including distribution-shifted challenge sets) and introduce
a new large-scale dataset based on texts archived by Project Gutenberg.
Surprisingly, we find that a traditional Ngram-based model performs best on 5
(of 7) AA tasks, achieving an average macro-accuracy of $76.50\%$ (compared to
$66.71\%$ for a BERT-based model). However, on the two AA datasets with the
greatest number of words per author, as well as on the AV datasets, BERT-based
models perform best. While AV methods are easily applied to AA, they are seldom
included as baselines in AA papers. We show that through the application of
hard-negative mining, AV methods are competitive alternatives to AA methods.
Valla and all experiment code can be found here:
https://github.com/JacobTyo/Valla
