  There is a growing interest in the combined use of NLP and machine learning
methods to predict gaze patterns during naturalistic reading. While promising
results have been obtained through the use of transformer-based language
models, little work has been undertaken to relate the performance of such
models to general text characteristics. In this paper we report on experiments
with two eye-tracking corpora of naturalistic reading and two language models
(BERT and GPT-2). In all experiments, we test effects of a broad spectrum of
features for predicting human reading behavior that fall into five categories
(syntactic complexity, lexical richness, register-based multiword combinations,
readability and psycholinguistic word properties). Our experiments show that
both the features included and the architecture of the transformer-based
language models play a role in predicting multiple eye-tracking measures during
naturalistic reading. We also report the results of experiments aimed at
determining the relative importance of features from different groups using
SP-LIME.
