  Word-by-word conditional probabilities from Transformer-based language models
are increasingly being used to evaluate their predictions over minimal pairs or
to model the incremental processing difficulty of human readers. In this paper,
we argue that there is a confound posed by the subword tokenization scheme of
such language models, which has gone unaddressed thus far. This is due to the
fact that tokens in the subword vocabulary of most language models have leading
whitespaces and therefore do not naturally define stop probabilities of words.
We first prove that this can result in word probabilities that sum to more than
one, thereby violating the axiom that $\mathsf{P}(\Omega) = 1$. This property
results in a misallocation of word-by-word surprisal, where the unacceptability
of the current 'end of word' is incorrectly carried over to the next word.
Additionally, language models' such implicit prediction of word boundaries is
incongruous with psycholinguistic experiments where human subjects directly
observe upcoming word boundaries. We present a simple decoding technique to
reaccount the probability of the trailing whitespace into that of the current
word, which resolves this confound. As a case study, we show that this results
in significantly different estimates of garden-path effects in
transitive/intransitive sentences, where a comma is strongly expected before
the critical word.
