  Language models (LMs) estimate the probability distribution over sequences of
natural language; these distributions are crucial for computing perplexity and
surprisal in linguistics research. While we are usually concerned with
measuring these values for words, most LMs operate over subwords. Despite
seemingly straightforward, accurately computing probabilities over one unit
given probabilities over the other requires care. Indeed, we show here that
many recent linguistic studies have been incorrectly computing these values.
This paper derives the correct methods for computing word probabilities,
highlighting issues when relying on language models that use beginning-of-word
(bow)-marking tokenisers, e.g., the GPT family. Empirically, we show that
correcting the widespread bug in probability computations affects measured
outcomes in sentence comprehension and lexical optimisation analyses.
