  Medical texts are notoriously challenging to read. Properly measuring their
readability is the first step towards making them more accessible. In this
paper, we present a systematic study on fine-grained readability measurements
in the medical domain at both sentence-level and span-level. We introduce a new
dataset MedReadMe, which consists of manually annotated readability ratings and
fine-grained complex span annotation for 4,520 sentences, featuring two novel
"Google-Easy" and "Google-Hard" categories. It supports our quantitative
analysis, which covers 650 linguistic features and automatic complex word and
jargon identification. Enabled by our high-quality annotation, we benchmark and
improve several state-of-the-art sentence-level readability metrics for the
medical domain specifically, which include unsupervised, supervised, and
prompting-based methods using recently developed large language models (LLMs).
Informed by our fine-grained complex span annotation, we find that adding a
single feature, capturing the number of jargon spans, into existing readability
formulas can significantly improve their correlation with human judgments. We
will publicly release the dataset and code.
