  Joint Embedding Self-Supervised Learning (JE-SSL) has seen rapid developments
in recent years, due to its promise to effectively leverage large unlabeled
data. The development of JE-SSL methods was driven primarily by the search for
ever increasing downstream classification accuracies, using huge computational
resources, and typically built upon insights and intuitions inherited from a
close parent JE-SSL method. This has led unwittingly to numerous pre-conceived
ideas that carried over across methods e.g. that SimCLR requires very large
mini batches to yield competitive accuracies; that strong and computationally
slow data augmentations are required. In this work, we debunk several such
ill-formed a priori ideas in the hope to unleash the full potential of JE-SSL
free of unnecessary limitations. In fact, when carefully evaluating
performances across different downstream tasks and properly optimizing
hyper-parameters of the methods, we most often -- if not always -- see that
these widespread misconceptions do not hold. For example we show that it is
possible to train SimCLR to learn useful representations, while using a single
image patch as negative example, and simple Gaussian noise as the only data
augmentation for the positive pair. Along these lines, in the hope to
democratize JE-SSL and to allow researchers to easily make more extensive
evaluations of their methods, we introduce an optimized PyTorch library for
SSL.
