  While there has been significant development of models for Plain Language
Summarization (PLS), evaluation remains a challenge. PLS lacks a dedicated
assessment metric, and the suitability of text generation evaluation metrics is
unclear due to the unique transformations involved (e.g., adding background
explanations, removing jargon). To address these questions, our study
introduces a granular meta-evaluation testbed, APPLS, designed to evaluate
metrics for PLS. We identify four PLS criteria from previous work --
informativeness, simplification, coherence, and faithfulness -- and define a
set of perturbations corresponding to these criteria that sensitive metrics
should be able to detect. We apply these perturbations to extractive hypotheses
for two PLS datasets to form our testbed. Using APPLS, we assess performance of
14 metrics, including automated scores, lexical features, and LLM prompt-based
evaluations. Our analysis reveals that while some current metrics show
sensitivity to specific criteria, no single method captures all four criteria
simultaneously. We therefore recommend a suite of automated metrics be used to
capture PLS quality along all relevant criteria. This work contributes the
first meta-evaluation testbed for PLS and a comprehensive evaluation of
existing metrics. APPLS and our evaluation code is available at
https://github.com/LinguisticAnomalies/APPLS.
