  Pre-training models on Imagenet or other massive datasets of real images has
led to major advances in computer vision, albeit accompanied with shortcomings
related to curation cost, privacy, usage rights, and ethical issues. In this
paper, for the first time, we study the transferability of pre-trained models
based on synthetic data generated by graphics simulators to downstream tasks
from very different domains. In using such synthetic data for pre-training, we
find that downstream performance on different tasks are favored by different
configurations of simulation parameters (e.g. lighting, object pose,
backgrounds, etc.), and that there is no one-size-fits-all solution. It is thus
better to tailor synthetic pre-training data to a specific downstream task, for
best performance. We introduce Task2Sim, a unified model mapping downstream
task representations to optimal simulation parameters to generate synthetic
pre-training data for them. Task2Sim learns this mapping by training to find
the set of best parameters on a set of "seen" tasks. Once trained, it can then
be used to predict best simulation parameters for novel "unseen" tasks in one
shot, without requiring additional training. Given a budget in number of images
per class, our extensive experiments with 20 diverse downstream tasks show
Task2Sim's task-adaptive pre-training data results in significantly better
downstream performance than non-adaptively choosing simulation parameters on
both seen and unseen tasks. It is even competitive with pre-training on real
images from Imagenet.
