  Ultrasound imaging is a commonly used technology for visualising patient
anatomy in real-time during diagnostic and therapeutic procedures. High
operator dependency and low reproducibility make ultrasound imaging and
interpretation challenging with a steep learning curve. Automatic image
classification using deep learning has the potential to overcome some of these
challenges by supporting ultrasound training in novices, as well as aiding
ultrasound image interpretation in patient with complex pathology for more
experienced practitioners. However, the use of deep learning methods requires a
large amount of data in order to provide accurate results. Labelling large
ultrasound datasets is a challenging task because labels are retrospectively
assigned to 2D images without the 3D spatial context available in vivo or that
would be inferred while visually tracking structures between frames during the
procedure. In this work, we propose a multi-modal convolutional neural network
(CNN) architecture that labels endoscopic ultrasound (EUS) images from raw
verbal comments provided by a clinician during the procedure. We use a CNN
composed of two branches, one for voice data and another for image data, which
are joined to predict image labels from the spoken names of anatomical
landmarks. The network was trained using recorded verbal comments from expert
operators. Our results show a prediction accuracy of 76% at image level on a
dataset with 5 different labels. We conclude that the addition of spoken
commentaries can increase the performance of ultrasound image classification,
and eliminate the burden of manually labelling large EUS datasets necessary for
deep learning applications.
