  Self-supervised learning (SSL) methods are popular since they can address
situations with limited annotated data by directly utilising the underlying
data distribution. However, the adoption of such methods is not explored enough
in ultrasound (US) imaging, especially for fetal assessment. We investigate the
potential of dual-encoder SSL in utilizing unlabelled US video data to improve
the performance of challenging downstream Standard Fetal Cardiac Planes (SFCP)
classification using limited labelled 2D US images. We study 7 SSL approaches
based on reconstruction, contrastive loss, distillation, and information theory
and evaluate them extensively on a large private US dataset. Our observations
and findings are consolidated from more than 500 downstream training
experiments under different settings. Our primary observation shows that for
SSL training, the variance of the dataset is more crucial than its size because
it allows the model to learn generalisable representations, which improve the
performance of downstream tasks. Overall, the BarlowTwins method shows robust
performance, irrespective of the training settings and data variations, when
used as an initialisation for downstream tasks. Notably, full fine-tuning with
1% of labelled data outperforms ImageNet initialisation by 12% in F1-score and
outperforms other SSL initialisations by at least 4% in F1-score, thus making
it a promising candidate for transfer learning from US video to image data.
