  Neural language models typically tokenise input text into sub-word units to
achieve an open vocabulary. The standard approach is to use a single canonical
tokenisation at both train and test time. We suggest that this approach is
unsatisfactory and may bottleneck our evaluation of language model performance.
Using only the one-best tokenisation ignores tokeniser uncertainty over
alternative tokenisations, which may hurt model out-of-domain performance.
  In this paper, we argue that instead, language models should be evaluated on
their marginal likelihood over tokenisations. We compare different estimators
for the marginal likelihood based on sampling, and show that it is feasible to
estimate the marginal likelihood with a manageable number of samples. We then
evaluate pretrained English and German language models on both the
one-best-tokenisation and marginal perplexities, and show that the marginal
perplexity can be significantly better than the one best, especially on
out-of-domain data. We link this difference in perplexity to the tokeniser
uncertainty as measured by tokeniser entropy. We discuss some implications of
our results for language model training and evaluation, particularly with
regard to tokenisation robustness.
