  Unsupervised transfer of object recognition models from synthetic to real
data is an important problem with many potential applications. The challenge is
how to "adapt" a model trained on simulated images so that it performs well on
real-world data without any additional supervision. Unfortunately, current
benchmarks for this problem are limited in size and task diversity. In this
paper, we present a new large-scale benchmark called Syn2Real, which consists
of a synthetic domain rendered from 3D object models and two real-image domains
containing the same object categories. We define three related tasks on this
benchmark: closed-set object classification, open-set object classification,
and object detection. Our evaluation of multiple state-of-the-art methods
reveals a large gap in adaptation performance between the easier closed-set
classification task and the more difficult open-set and detection tasks. We
conclude that developing adaptation methods that work well across all three
tasks presents a significant future challenge for syn2real domain transfer.
