  Self-Supervised learning (SSL) with Joint-Embedding Architectures (JEA) has
led to outstanding performances. All instantiations of this paradigm were
trained using strong and well-established hand-crafted data augmentations,
leading to the general belief that they are required for the proper training
and performance of such models. On the other hand, generative
reconstruction-based models such as BEIT and MAE or Joint-Embedding Predictive
Architectures such as I-JEPA have shown strong performance without using data
augmentations except masking. In this work, we challenge the importance of
invariance and data-augmentation in JEAs at scale. By running a case-study on a
recent SSL foundation model - DINOv2 - we show that strong image
representations can be obtained with JEAs and only cropping without resizing
provided the training data is large enough, reaching state-of-the-art results
and using the least amount of augmentation in the literature. Through this
study, we also discuss the impact of compute constraints on the outcomes of
experimental deep learning research, showing that they can lead to very
different conclusions.
