  Readability assessment aims to automatically classify text by the level
appropriate for learning readers. Traditional approaches to this task utilize a
variety of linguistically motivated features paired with simple machine
learning models. More recent methods have improved performance by discarding
these features and utilizing deep learning models. However, it is unknown
whether augmenting deep learning models with linguistically motivated features
would improve performance further. This paper combines these two approaches
with the goal of improving overall model performance and addressing this
question. Evaluating on two large readability corpora, we find that, given
sufficient training data, augmenting deep learning models with linguistically
motivated features does not improve state-of-the-art performance. Our results
provide preliminary evidence for the hypothesis that the state-of-the-art deep
learning models represent linguistic features of the text related to
readability. Future research on the nature of representations formed in these
models can shed light on the learned features and their relations to
linguistically motivated ones hypothesized in traditional approaches.
