  The current literature on self-supervised learning (SSL) focuses on
developing learning objectives to train neural networks more effectively on
unlabeled data. The typical development process involves taking
well-established architectures, e.g., ResNet demonstrated on ImageNet, and
using them to evaluate newly developed objectives on downstream scenarios.
While convenient, this does not take into account the role of architectures
which has been shown to be crucial in the supervised learning literature. In
this work, we establish extensive empirical evidence showing that a network
architecture plays a significant role in SSL. We conduct a large-scale study
with over 100 variants of ResNet and MobileNet architectures and evaluate them
across 11 downstream scenarios in the SSL setting. We show that there is no one
network that performs consistently well across the scenarios. Based on this, we
propose to learn not only network weights but also architecture topologies in
the SSL regime. We show that "self-supervised architectures" outperform popular
handcrafted architectures (ResNet18 and MobileNetV2) while performing
competitively with the larger and computationally heavy ResNet50 on major image
classification benchmarks (ImageNet-1K, iNat2021, and more). Our results
suggest that it is time to consider moving beyond handcrafted architectures in
SSL and start thinking about incorporating architecture search into
self-supervised learning objectives.
