  Clinician must write a lengthy summary each time a patient is discharged from
the hospital. This task is time-consuming due to the sheer number of unique
clinical concepts covered in the admission. Identifying and covering salient
entities is vital for the summary to be clinically useful. We fine-tune
open-source LLMs (Mistral-7B-Instruct and Zephyr-7B-\b{eta}) on the task and
find that they generate incomplete and unfaithful summaries. To increase entity
coverage, we train a smaller, encoder-only model to predict salient entities,
which are treated as content-plans to guide the LLM. To encourage the LLM to
focus on specific mentions in the source notes, we propose SPEER:
Sentence-level Planning via Embedded Entity Retrieval. Specifically, we mark
each salient entity span with special "{{ }}" boundary tags and instruct the
LLM to retrieve marked spans before generating each sentence. Sentence-level
planning acts as a form of state tracking in that the model is explicitly
recording the entities it uses. We fine-tune Mistral and Zephyr variants on a
large-scale, diverse dataset of ~167k in-patient hospital admissions and
evaluate on 3 datasets. SPEER shows gains in both coverage and faithfulness
metrics over non-guided and guided baselines.
