  Extracting patient information from unstructured text is a critical task in
health decision-support and clinical research. Large language models (LLMs)
have shown the potential to accelerate clinical curation via few-shot
in-context learning, in contrast to supervised learning which requires much
more costly human annotations. However, despite drastic advances in modern LLMs
such as GPT-4, they still struggle with issues regarding accuracy and
interpretability, especially in mission-critical domains such as health. Here,
we explore a general mitigation framework using self-verification, which
leverages the LLM to provide provenance for its own extraction and check its
own outputs. This is made possible by the asymmetry between verification and
generation, where the latter is often much easier than the former. Experimental
results show that our method consistently improves accuracy for various LLMs in
standard clinical information extraction tasks. Additionally, self-verification
yields interpretations in the form of a short text span corresponding to each
output, which makes it very efficient for human experts to audit the results,
paving the way towards trustworthy extraction of clinical information in
resource-constrained scenarios. To facilitate future research in this
direction, we release our code and prompts.
