  The ultrasound characteristics of thyroid nodules guide the evaluation of
thyroid cancer in patients with thyroid nodules. However, the characteristics
of thyroid nodules are often documented in clinical narratives such as
ultrasound reports. Previous studies have examined natural language processing
(NLP) methods in extracting a limited number of characteristics (<9) using
rule-based NLP systems. In this study, a multidisciplinary team of NLP experts
and thyroid specialists, identified thyroid nodule characteristics that are
important for clinical care, composed annotation guidelines, developed a
corpus, and compared 5 state-of-the-art transformer-based NLP methods,
including BERT, RoBERTa, LongFormer, DeBERTa, and GatorTron, for extraction of
thyroid nodule characteristics from ultrasound reports. Our GatorTron model, a
transformer-based large language model trained using over 90 billion words of
text, achieved the best strict and lenient F1-score of 0.8851 and 0.9495 for
the extraction of a total number of 16 thyroid nodule characteristics, and
0.9321 for linking characteristics to nodules, outperforming other clinical
transformer models. To the best of our knowledge, this is the first study to
systematically categorize and apply transformer-based NLP models to extract a
large number of clinical relevant thyroid nodule characteristics from
ultrasound reports. This study lays ground for assessing the documentation
quality of thyroid ultrasound reports and examining outcomes of patients with
thyroid nodules using electronic health records.
