  In this paper, we explore the possibility of building a unified foundation
model that can be adapted to both vision-only and text-only tasks. Starting
from BERT and ViT, we design a unified transformer consisting of
modality-specific tokenizers, a shared transformer encoder, and task-specific
output heads. To efficiently pre-train the proposed model jointly on unpaired
images and text, we propose two novel techniques: (i) We employ the
separately-trained BERT and ViT models as teachers and apply knowledge
distillation to provide additional, accurate supervision signals for the joint
training; (ii) We propose a novel gradient masking strategy to balance the
parameter updates from the image and text pre-training losses. We evaluate the
jointly pre-trained transformer by fine-tuning it on image classification tasks
and natural language understanding tasks, respectively. The experiments show
that the resultant unified foundation transformer works surprisingly well on
both the vision-only and text-only tasks, and the proposed knowledge
distillation and gradient masking strategy can effectively lift the performance
to approach the level of separately-trained models.
