  Rich temporal information and variations in viewpoints make video data an
attractive choice for learning image representations using unsupervised
contrastive learning (UCL) techniques. State-of-the-art (SOTA) contrastive
learning techniques consider frames within a video as positives in the
embedding space, whereas the frames from other videos are considered negatives.
We observe that unlike multiple views of an object in natural scene videos, an
Ultrasound (US) video captures different 2D slices of an organ. Hence, there is
almost no similarity between the temporally distant frames of even the same US
video. In this paper we propose to instead utilize such frames as hard
negatives. We advocate mining both intra-video and cross-video negatives in a
hardness-sensitive negative mining curriculum in a UCL framework to learn rich
image representations. We deploy our framework to learn the representations of
Gallbladder (GB) malignancy from US videos. We also construct the first
large-scale US video dataset containing 64 videos and 15,800 frames for
learning GB representations. We show that the standard ResNet50 backbone
trained with our framework improves the accuracy of models pretrained with SOTA
UCL techniques as well as supervised pretrained models on ImageNet for the GB
malignancy detection task by 2-6%. We further validate the generalizability of
our method on a publicly available lung US image dataset of COVID-19
pathologies and show an improvement of 1.5% compared to SOTA. Source code,
dataset, and models are available at https://gbc-iitd.github.io/usucl.
