  The goal of this work is to build a classifier that can identify text
complexity within the context of teaching reading to English as a Second
Language (ESL) learners. To present language learners with texts that are
suitable to their level of English, a set of features that can describe the
phonological, morphological, lexical, syntactic, discursive, and psychological
complexity of a given text were identified. Using a corpus of 6171 texts, which
had already been classified into three different levels of difficulty by ESL
experts, different experiments were conducted with five machine learning
algorithms. The results showed that the adopted linguistic features provide a
good overall classification performance (F-Score = 0.97). A scalability
evaluation was conducted to test if such a classifier could be used within real
applications, where it can be, for example, plugged into a search engine or a
web-scraping module. In this evaluation, the texts in the test set are not only
different from those from the training set but also of different types (ESL
texts vs. children reading texts). Although the overall performance of the
classifier decreased significantly (F-Score = 0.65), the confusion matrix shows
that most of the classification errors are between the classes two and three
(the middle-level classes) and that the system has a robust performance in
categorizing texts of class one and four. This behavior can be explained by the
difference in classification criteria between the two corpora. Hence, the
observed results confirm the usability of such a classifier within a real-world
application.
