  We present an effective method for fusing visual-and-language representations
for several question answering tasks including visual question answering and
visual entailment. In contrast to prior works that concatenate unimodal
representations or use only cross-attention, we compose multimodal
representations via channel fusion. By fusing on the channels, the model is
able to more effectively align the tokens compared to standard methods. These
multimodal representations, which we call compound tokens are generated with
cross-attention transformer layers. First, vision tokens are used as queries to
retrieve compatible text tokens through cross-attention. We then chain the
vision tokens and the queried text tokens along the channel dimension. We call
the resulting representations compound tokens. A second group of compound
tokens are generated using an analogous process where the text tokens serve as
queries to the cross-attention layer. We concatenate all the compound tokens
for further processing with multimodal encoder. We demonstrate the
effectiveness of compound tokens using an encoder-decoder vision-language model
trained end-to-end in the open-vocabulary setting. Compound Tokens achieve
highly competitive performance across a range of question answering tasks
including GQA, VQA2.0, and SNLI-VE.
