  In this paper, we identify an important reproducibility challenge in the
image-to-set prediction literature that impedes proper comparisons among
published methods, namely, researchers use different evaluation protocols to
assess their contributions. To alleviate this issue, we introduce an
image-to-set prediction benchmark suite built on top of five public datasets of
increasing task complexity that are suitable for multi-label classification
(VOC, COCO, NUS-WIDE, ADE20k and Recipe1M). Using the benchmark, we provide an
in-depth analysis where we study the key components of current models, namely
the choice of the image representation backbone as well as the set predictor
design. Our results show that (1) exploiting better image representation
backbones leads to higher performance boosts than enhancing set predictors, and
(2) modeling both the label co-occurrences and ordering has a slight positive
impact in terms of performance, whereas explicit cardinality prediction only
helps when training on complex datasets, such as Recipe1M. To facilitate future
image-to-set prediction research, we make the code, best models and dataset
splits publicly available at: https://github.com/facebookresearch/image-to-set.
