  Text simplification refers to the process of increasing the comprehensibility
of texts. Automatic text simplification models are most commonly evaluated by
experts or crowdworkers instead of the primary target groups of simplified
texts, such as persons with intellectual disabilities. We conducted an
evaluation study of text comprehensibility including participants with and
without intellectual disabilities reading unsimplified, automatically and
manually simplified German texts on a tablet computer. We explored four
different approaches to measuring comprehensibility: multiple-choice
comprehension questions, perceived difficulty ratings, response time, and
reading speed. The results revealed significant variations in these
measurements, depending on the reader group and whether the text had undergone
automatic or manual simplification. For the target group of persons with
intellectual disabilities, comprehension questions emerged as the most reliable
measure, while analyzing reading speed provided valuable insights into
participants' reading behavior.
