  The problem of accurately predicting relative reading difficulty across a set
of sentences arises in a number of important natural language applications,
such as finding and curating effective usage examples for intelligent language
tutoring systems. Yet while significant research has explored document- and
passage-level reading difficulty, the special challenges involved in assessing
aspects of readability for single sentences have received much less attention,
particularly when considering the role of surrounding passages. We introduce
and evaluate a novel approach for estimating the relative reading difficulty of
a set of sentences, with and without surrounding context. Using different sets
of lexical and grammatical features, we explore models for predicting pairwise
relative difficulty using logistic regression, and examine rankings generated
by aggregating pairwise difficulty labels using a Bayesian rating system to
form a final ranking. We also compare rankings derived for sentences assessed
with and without context, and find that contextual features can help predict
differences in relative difficulty judgments across these two conditions.
