  The rapid adoption of Electronic Health Records (EHRs) has been instrumental
in streamlining administrative tasks, increasing transparency, and enabling
continuity of care across providers. An unintended consequence of the increased
documentation burden, however, has been reduced face-time with patients and,
concomitantly, a dramatic rise in clinician burnout. In this thesis, we
pinpoint a particularly time-intensive, yet critical, documentation task:
generating a summary of a patient's hospital admissions, and propose and
evaluate automated solutions. In Chapter 2, we construct a dataset based on
109,000 hospitalizations (2M source notes) and perform exploratory analyses to
motivate future work on modeling and evaluation [NAACL 2021]. In Chapter 3, we
address faithfulness from a modeling perspective by revising noisy references
[EMNLP 2022] and, to reduce the reliance on references, directly calibrating
model outputs to metrics [ACL 2023]. These works relied heavily on automatic
metrics as human annotations were limited. To fill this gap, in Chapter 4, we
conduct a fine-grained expert annotation of system errors in order to
meta-evaluate existing metrics and better understand task-specific issues of
domain adaptation and source-summary alignments. To learn a metric less
correlated to extractiveness (copy-and-paste), we derive noisy faithfulness
labels from an ensemble of existing metrics and train a faithfulness classifier
on these pseudo labels [MLHC 2023]. Finally, in Chapter 5, we demonstrate that
fine-tuned LLMs (Mistral and Zephyr) are highly prone to entity hallucinations
and cover fewer salient entities. We improve both coverage and faithfulness by
performing sentence-level entity planning based on a set of pre-computed
salient entities from the source text, which extends our work on entity-guided
news summarization [ACL, 2023], [EMNLP, 2023].
