  Artificial neural networks trained on large, expert-labelled datasets are
considered state-of-the-art for a range of medical image recognition tasks.
However, categorically labelled datasets are time-consuming to generate and
constrain classification to a pre-defined, fixed set of classes. For
neuroradiological applications in particular, this represents a barrier to
clinical adoption. To address these challenges, we present a self-supervised
text-vision framework that learns to detect clinically relevant abnormalities
in brain MRI scans by directly leveraging the rich information contained in
accompanying free-text neuroradiology reports. Our training approach consisted
of two-steps. First, a dedicated neuroradiological language model - NeuroBERT -
was trained to generate fixed-dimensional vector representations of
neuroradiology reports (N = 50,523) via domain-specific self-supervised
learning tasks. Next, convolutional neural networks (one per MRI sequence)
learnt to map individual brain scans to their corresponding text vector
representations by optimising a mean square error loss. Once trained, our
text-vision framework can be used to detect abnormalities in unreported brain
MRI examinations by scoring scans against suitable query sentences (e.g.,
'there is an acute stroke', 'there is hydrocephalus' etc.), enabling a range of
classification-based applications including automated triage. Potentially, our
framework could also serve as a clinical decision support tool, not only by
suggesting findings to radiologists and detecting errors in provisional
reports, but also by retrieving and displaying examples of pathologies from
historical examinations that could be relevant to the current case based on
textual descriptors.
