  The paradigm of training models on massive data without label through
self-supervised learning (SSL) and finetuning on many downstream tasks has
become a trend recently. However, due to the high training costs and the
unconsciousness of downstream usages, most self-supervised learning methods
lack the capability to correspond to the diversities of downstream scenarios,
as there are various data domains, different vision tasks and latency
constraints on models. Neural architecture search (NAS) is one universally
acknowledged fashion to conquer the issues above, but applying NAS on SSL seems
impossible as there is no label or metric provided for judging model selection.
In this paper, we present DATA, a simple yet effective NAS approach specialized
for SSL that provides Domain-Aware and Task-Aware pre-training. Specifically,
we (i) train a supernet which could be deemed as a set of millions of networks
covering a wide range of model scales without any label, (ii) propose a
flexible searching mechanism compatible with SSL that enables finding networks
of different computation costs, for various downstream vision tasks and data
domains without explicit metric provided. Instantiated With MoCo v2, our method
achieves promising results across a wide range of computation costs on
downstream tasks, including image classification, object detection and semantic
segmentation. DATA is orthogonal to most existing SSL methods and endows them
the ability of customization on downstream needs. Extensive experiments on
other SSL methods demonstrate the generalizability of the proposed method. Code
is released at https://github.com/GAIA-vision/GAIA-ssl
