  This paper investigates the feasibility of using pre-trained generative Large
Language Models (LLMs) to automate the assignment of ICD-10 codes to historical
causes of death. Due to the complex narratives often found in historical causes
of death, this task has traditionally been manually performed by coding
experts. We evaluate the ability of GPT-3.5, GPT-4, and Llama 2 LLMs to
accurately assign ICD-10 codes on the HiCaD dataset that contains causes of
death recorded in the civil death register entries of 19,361 individuals from
Ipswich, Kilmarnock, and the Isle of Skye from the UK between 1861-1901. Our
findings show that GPT-3.5, GPT-4, and Llama 2 assign the correct code for 69%,
83%, and 40% of causes, respectively. However, we achieve a maximum accuracy of
89% by standard machine learning techniques. All LLMs performed better for
causes of death that contained terms still in use today, compared to archaic
terms. Also they perform better for short causes (1-2 words) compared to longer
causes. LLMs therefore do not currently perform well enough for historical
ICD-10 code assignment tasks. We suggest further fine-tuning or alternative
frameworks to achieve adequate performance.
