  For centuries, writers have hidden messages in their texts as acrostics,
where initial letters of consecutive lines or paragraphs form meaningful words
or phrases. Scholars searching for acrostics manually can only focus on a few
authors at a time and often favor qualitative arguments in discussing
intentionally. We aim to put the study of acrostics on firmer statistical
footing by presenting AcrosticSleuth, a first-of-its-kind tool that
automatically identifies acrostics and ranks them by the probability that the
sequence of characters does not occur by chance (and therefore may have been
inserted intentionally). Acrostics are rare, so we formalize the problem as a
binary classification task in the presence of extreme class imbalance. To
evaluate AcrosticSleuth, we present the Acrostic Identification Dataset
(AcrostID), a collection of acrostics from the WikiSource online database.
Despite the class imbalance, AcrosticSleuth achieves F1 scores of 0.39, 0.59,
and 0.66 on French, English, and Russian subdomains of WikiSource,
respectively. We further demonstrate that AcrosticSleuth can identify
previously unknown high-profile instances of wordplay, such as the acrostic
spelling ARSPOETICA (``art of poetry") by Italian Humanist Albertino Mussato
and English philosopher Thomas Hobbes' signature in the opening paragraphs of
The Elements of Law.
