  The ability to model intra-modal and inter-modal interactions is fundamental
in multimodal machine learning. The current state-of-the-art models usually
adopt deep learning models with fixed structures. They can achieve exceptional
performances on specific tasks, but face a particularly challenging problem of
modality mismatch because of diversity of input modalities and their fixed
structures. In this paper, we present \textbf{Switch-BERT} for joint vision and
language representation learning to address this problem. Switch-BERT extends
BERT architecture by introducing learnable layer-wise and cross-layer
interactions. It learns to optimize attention from a set of attention modes
representing these interactions. One specific property of the model is that it
learns to attend outputs from various depths, therefore mitigates the modality
mismatch problem. We present extensive experiments on visual question
answering, image-text retrieval and referring expression comprehension
experiments. Results confirm that, whereas alternative architectures including
ViLBERT and UNITER may excel in particular tasks, Switch-BERT can consistently
achieve better or comparable performances than the current state-of-the-art
models in these tasks. Ablation studies indicate that the proposed model
achieves superior performances due to its ability in learning task-specific
multimodal interactions.
