  BACKGROUND: Radiology reports are typically written in a free-text format,
making clinical information difficult to extract and use. Recently the adoption
of structured reporting (SR) has been recommended by various medical societies
thanks to the advantages it offers, e.g. standardization, completeness and
information retrieval. We propose a pipeline to extract information from
free-text radiology reports, that fits with the items of the reference SR
registry proposed by a national society of interventional and medical
radiology, focusing on CT staging of patients with lymphoma. METHODS: Our work
aims to leverage the potential of Natural Language Processing (NLP) and
Transformer-based models to deal with automatic SR registry filling. With the
availability of 174 radiology reports, we investigate a rule-free generative
Question Answering approach based on a domain-specific version of T5 (IT5). Two
strategies (batch-truncation and ex-post combination) are implemented to comply
with the model's context length limitations. Performance is evaluated in terms
of strict accuracy, F1, and format accuracy, and compared with the widely used
GPT-3.5 Large Language Model. A 5-point Likert scale questionnaire is used to
collect human-expert feedback on the similarity between medical annotations and
generated answers. RESULTS: The combination of fine-tuning and batch splitting
allows IT5 to achieve notable results; it performs on par with GPT-3.5 albeit
its size being a thousand times smaller in terms of parameters. Human-based
assessment scores show a high correlation (Spearman's correlation
coefficients>0.88, p-values<0.001) with AI performance metrics (F1) and confirm
the superior ability of LLMs (i.e., GPT-3.5, 175B of parameters) in generating
plausible human-like statements.
