  Sentence simplification, which rewrites a sentence to be easier to read and
understand, is a promising technique to help people with various reading
difficulties. With the rise of advanced large language models (LLMs),
evaluating their performance in sentence simplification has become imperative.
Recent studies have used both automatic metrics and human evaluations to assess
the simplification abilities of LLMs. However, the suitability of existing
evaluation methodologies for LLMs remains in question. First, the suitability
of current automatic metrics on LLMs' simplification evaluation is still
uncertain. Second, current human evaluation approaches in sentence
simplification often fall into two extremes: they are either too superficial,
failing to offer a clear understanding of the models' performance, or overly
detailed, making the annotation process complex and prone to inconsistency,
which in turn affects the evaluation's reliability. To address these problems,
this study provides in-depth insights into LLMs' performance while ensuring the
reliability of the evaluation. We design an error-based human annotation
framework to assess the GPT-4's simplification capabilities. Results show that
GPT-4 generally generates fewer erroneous simplification outputs compared to
the current state-of-the-art. However, LLMs have their limitations, as seen in
GPT-4's struggles with lexical paraphrasing. Furthermore, we conduct
meta-evaluations on widely used automatic metrics using our human annotations.
We find that while these metrics are effective for significant quality
differences, they lack sufficient sensitivity to assess the overall
high-quality simplification by GPT-4.
