  Learned self-attention functions in state-of-the-art NLP models often
correlate with human attention. We investigate whether self-attention in
large-scale pre-trained language models is as predictive of human eye fixation
patterns during task-reading as classical cognitive models of human attention.
We compare attention functions across two task-specific reading datasets for
sentiment analysis and relation extraction. We find the predictiveness of
large-scale pre-trained self-attention for human attention depends on `what is
in the tail', e.g., the syntactic nature of rare contexts. Further, we observe
that task-specific fine-tuning does not increase the correlation with human
task-specific reading. Through an input reduction experiment we give
complementary insights on the sparsity and fidelity trade-off, showing that
lower-entropy attention vectors are more faithful.
