  We propose a new method for evaluating the readability of simplified
sentences through pair-wise ranking. The validity of the method is established
through in-corpus and cross-corpus evaluation experiments. The approach
correctly identifies the ranking of simplified and unsimplified sentences in
terms of their reading level with an accuracy of over 80%, significantly
outperforming previous results. To gain qualitative insights into the nature of
simplification at the sentence level, we studied the impact of specific
linguistic features. We empirically confirm that both word-level and syntactic
features play a role in comparing the degree of simplification of authentic
data. To carry out this research, we created a new sentence-aligned corpus from
professionally simplified news articles. The new corpus resource enriches the
empirical basis of sentence-level simplification research, which so far relied
on a single resource. Most importantly, it facilitates cross-corpus evaluation
for simplification, a key step towards generalizable results.
