  An important assumption that comes with using LLMs on psycholinguistic data
has gone unverified. LLM-based predictions are based on subword tokenization,
not decomposition of words into morphemes. Does that matter? We carefully test
this by comparing surprisal estimates using orthographic, morphological, and
BPE tokenization against reading time data. Our results replicate previous
findings and provide evidence that in the aggregate, predictions using BPE
tokenization do not suffer relative to morphological and orthographic
segmentation. However, a finer-grained analysis points to potential issues with
relying on BPE-based tokenization, as well as providing promising results
involving morphologically-aware surprisal estimates and suggesting a new method
for evaluating morphological prediction.
