  Ultrasound (US) is one of the most common medical imaging modalities since it
is radiation-free, low-cost, and real-time. In freehand US examinations,
sonographers often navigate a US probe to visualize standard examination planes
with rich diagnostic information. However, reproducibility and stability of the
resulting images often suffer from intra- and inter-operator variation.
Reinforcement learning (RL), as an interaction-based learning method, has
demonstrated its effectiveness in visual navigating tasks; however, RL is
limited in terms of generalization. To address this challenge, we propose a
simulation-based RL framework for real-world navigation of US probes towards
the standard longitudinal views of vessels. A UNet is used to provide binary
masks from US images; thereby, the RL agent trained on simulated binary vessel
images can be applied in real scenarios without further training. To accurately
characterize actual states, a multi-modality state representation structure is
introduced to facilitate the understanding of environments. Moreover,
considering the characteristics of vessels, a novel standard view recognition
approach based on the minimum bounding rectangle is proposed to terminate the
searching process. To evaluate the effectiveness of the proposed method, the
trained policy is validated virtually on 3D volumes of a volunteer's in-vivo
carotid artery, and physically on custom-designed gel phantoms using robotic
US. The results demonstrate that proposed approach can effectively and
accurately navigate the probe towards the longitudinal view of vessels.
