  Transformer-based large language models are trained to make predictions about
the next word by aggregating representations of previous tokens through their
self-attention mechanism. In the field of cognitive modeling, such attention
patterns have recently been interpreted as embodying the process of cue-based
retrieval, in which attention over multiple targets is taken to generate
interference and latency during retrieval. Under this framework, this work
first defines an entropy-based predictor that quantifies the diffuseness of
self-attention, as well as distance-based predictors that capture the
incremental change in attention patterns across timesteps. Moreover, following
recent studies that question the informativeness of attention weights, we also
experiment with alternative methods for incorporating vector norms into
attention weights. Regression experiments using predictors calculated from the
GPT-2 language model show that these predictors deliver a substantially better
fit to held-out self-paced reading and eye-tracking data over a rigorous
baseline including GPT-2 surprisal. Additionally, the distance-based predictors
generally demonstrated higher predictive power, with effect sizes of up to 6.59
ms per standard deviation on self-paced reading times (compared to 2.82 ms for
surprisal) and 1.05 ms per standard deviation on eye-gaze durations (compared
to 3.81 ms for surprisal).
