  We present an automatic method to describe clinically useful information
about scanning, and to guide image interpretation in ultrasound (US) videos of
the fetal heart. Our method is able to jointly predict the visibility, viewing
plane, location and orientation of the fetal heart at the frame level. The
contributions of the paper are three-fold: (i) a convolutional neural network
architecture is developed for a multi-task prediction, which is computed by
sliding a 3x3 window spatially through convolutional maps. (ii) an anchor
mechanism and Intersection over Union (IoU) loss are applied for improving
localization accuracy. (iii) a recurrent architecture is designed to
recursively compute regional convolutional features temporally over sequential
frames, allowing each prediction to be conditioned on the whole video. This
results in a spatial-temporal model that precisely describes detailed heart
parameters in challenging US videos. We report results on a real-world clinical
dataset, where our method achieves performance on par with expert annotations.
