  While self-supervised pretraining has proven beneficial for many computer
vision tasks, it requires expensive and lengthy computation, large amounts of
data, and is sensitive to data augmentation. Prior work demonstrates that
models pretrained on datasets dissimilar to their target data, such as chest
X-ray models trained on ImageNet, underperform models trained from scratch.
Users that lack the resources to pretrain must use existing models with lower
performance. This paper explores Hierarchical PreTraining (HPT), which
decreases convergence time and improves accuracy by initializing the
pretraining process with an existing pretrained model. Through experimentation
on 16 diverse vision datasets, we show HPT converges up to 80x faster, improves
accuracy across tasks, and improves the robustness of the self-supervised
pretraining process to changes in the image augmentation policy or amount of
pretraining data. Taken together, HPT provides a simple framework for obtaining
better pretrained representations with less computational resources.
