  A large labeled dataset is a key to the success of supervised deep learning,
but for medical image segmentation, it is highly challenging to obtain
sufficient annotated images for model training. In many scenarios, unannotated
images are abundant and easy to acquire. Self-supervised learning (SSL) has
shown great potentials in exploiting raw data information and representation
learning. In this paper, we propose Hierarchical Self-Supervised Learning
(HSSL), a new self-supervised framework that boosts medical image segmentation
by making good use of unannotated data. Unlike the current literature on
task-specific self-supervised pretraining followed by supervised fine-tuning,
we utilize SSL to learn task-agnostic knowledge from heterogeneous data for
various medical image segmentation tasks. Specifically, we first aggregate a
dataset from several medical challenges, then pre-train the network in a
self-supervised manner, and finally fine-tune on labeled data. We develop a new
loss function by combining contrastive loss and classification loss and
pretrain an encoder-decoder architecture for segmentation tasks. Our extensive
experiments show that multi-domain joint pre-training benefits downstream
segmentation tasks and outperforms single-domain pre-training significantly.
Compared to learning from scratch, our new method yields better performance on
various tasks (e.g., +0.69% to +18.60% in Dice scores with 5% of annotated
data). With limited amounts of training data, our method can substantially
bridge the performance gap w.r.t. denser annotations (e.g., 10% vs.~100% of
annotated data).
