  Ultrasound scanning is essential in several medical diagnostic and
therapeutic applications. It is used to visualize and analyze anatomical
features and structures that influence treatment plans. However, it is both
labor intensive, and its effectiveness is operator dependent. Real-time
accurate and robust automatic detection and tracking of anatomical structures
while scanning would significantly impact diagnostic and therapeutic procedures
to be consistent and efficient. In this paper, we propose a deep learning
framework to automatically detect and track a specific anatomical target
structure in ultrasound scans. Our framework is designed to be accurate and
robust across subjects and imaging devices, to operate in real-time, and to not
require a large training set. It maintains a localization precision and recall
higher than 90% when trained on training sets that are as small as 20% in size
of the original training set. The framework backbone is a weakly trained
segmentation neural network based on U-Net. We tested the framework on two
different ultrasound datasets with the aim to detect and track the Vagus nerve,
where it outperformed current state-of-the-art real-time object detection
networks.
