  Many Self-Supervised Learning (SSL) methods aim for model invariance to
different image augmentations known as views. To achieve this invariance,
conventional approaches make use of random sampling operations within the image
augmentation pipeline. We hypothesize that the efficacy of pretraining
pipelines based on conventional random view sampling can be enhanced by
explicitly selecting views that benefit the learning progress. A simple, yet
effective approach is to select hard views that yield a higher loss. In this
paper, we present Hard View Pretraining (HVP), a learning-free strategy that
builds upon this hypothesis and extends random view generation. HVP exposes the
model to harder, more challenging samples during SSL pretraining, which
enhances downstream performance. It encompasses the following iterative steps:
1) randomly sample multiple views and forward each view through the pretrained
model, 2) create pairs of two views and compute their loss, 3) adversarially
select the pair yielding the highest loss depending on the current model state,
and 4) run the backward pass with the selected pair. As a result, HVP achieves
linear evaluation accuracy improvements of 1% on average on ImageNet for both
100 and 300 epoch pretraining and similar improvements on transfer tasks across
DINO, SimSiam, iBOT, and SimCLR.
