  Most recent self-supervised learning methods learn visual representation by
contrasting different augmented views of images. Compared with supervised
learning, more aggressive augmentations have been introduced to further improve
the diversity of training pairs. However, aggressive augmentations may distort
images' structures leading to a severe semantic shift problem that augmented
views of the same image may not share the same semantics, thus degrading the
transfer performance. To address this problem, we propose a new SSL paradigm,
which counteracts the impact of semantic shift by balancing the role of weak
and aggressively augmented pairs. Specifically, semantically inconsistent pairs
are of minority and we treat them as noisy pairs. Note that deep neural
networks (DNNs) have a crucial memorization effect that DNNs tend to first
memorize clean (majority) examples before overfitting to noisy (minority)
examples. Therefore, we set a relatively large weight for aggressively
augmented data pairs at the early learning stage. With the training going on,
the model begins to overfit noisy pairs. Accordingly, we gradually reduce the
weights of aggressively augmented pairs. In doing so, our method can better
embrace the aggressive augmentations and neutralize the semantic shift problem.
Experiments show that our model achieves 73.1% top-1 accuracy on ImageNet-1K
with ResNet-50 for 200 epochs, which is a 2.5% improvement over BYOL. Moreover,
experiments also demonstrate that the learned representations can transfer well
for various downstream tasks.
