  Image-classification datasets have been used to pretrain image recognition
models. Recently, web-scale image-caption datasets have emerged as a source of
powerful pretraining alternative. Image-caption datasets are more
``open-domain'', containing a wider variety of scene types and vocabulary words
than traditional classification datasets, and models trained on these datasets
have demonstrated strong performance on few- and zero-shot recognition tasks.
When naively unifying image-classification and -caption dataset, we show that
such dataset biases negatively affect pre-training by reducing the
generalizability of learned representations and thus jeopardizing zero-shot
performance since the unification can tailor the model for the classification
dataset, making it vulnerable to the distribution shift from the dataset. In
this work, we address the problem by disentangling the dataset bias using
prefix tokens that inform a language encoder of the type of the input dataset
(e.g., image-classification or caption) at training time. This approach allows
the language encoder to share the knowledge from two datasets as well as switch
the mode of feature extraction, i.e., image-classification dataset or
image-caption dataset tailored mode, where we use image-caption mode in the
zero-shot evaluation. Our method is generic and can be easily integrated into
existing VL pre-training objectives such as CLIP or UniCL. In experiments, we
show that this simple technique improves the performance in zero-shot image
recognition accuracy and robustness to the image-level distribution shift.
