
\section{Introduction}

It has long been understood that the meaning of a word is systematically and predictably linked to the context in which it occurs (e.g., \citealt{firth1957synopsis,harris1954distributional,deerwester1990indexing,mikolov2013distributed}).  Different notions of context have resulted in different levels of success with downstream NLP tasks. Recent neural architectures including Transformer \citep{vaswani2017attention} and BERT \citep{devlin2018bert} have dramatically increased our ability to include a broad window of potential lexical hints.  However, the same capacity allows for multimodal context, which may help model the meaning of words in general, and also sharpen its understanding of instances of words in context (e.g., \citealt{bruni2014multimodal}).  

\begin{figure}
  \includegraphics[width=\columnwidth]{rationale-needed.png}
\begin{scriptsize}
{\setlength{\tabcolsep}{2pt}
\begin{tabular}{cp{6.5cm}} \\
Q: & What was [1] doing before he sat
   in his living room? \vspace{1mm} \\
A$_1$: & He was reading [10]. \\
A$_2$: & He was taking a shower. \checkmark \\
A$_3$: & [0] was sleeping until the noise 
    [1] was making woke him up.\\
A$_4$: & He was sleeping in his bedroom.
   \vspace{1mm} \\
R$_1$: & His clothes are disheveled and his
    face is glistening like he's sweaty. \\
R$_2$: & [0] does not look wet yet, but [0]
    looks like his hair is wet, and bathrobes
    are what you wear before or after a shower. \\
R$_3$: & He is still wearing his bathrobe. \checkmark\\
R$_4$: & His hair appears wet and there is clothing
    hanging in front of him on a line as if to dry.
\end{tabular}}
\end{scriptsize}
\caption{An example from the VCR dataset. The tasks consists in picking an answer A$_{1-4}$, and then picking a rationale R$_{1-4}$. The data contains explicit pointers in the text to bounding boxes in the image.}
\label{fig:rationale-needed}
\end{figure}

In this paper, we consider visual context in addition to language and show that the right integration of visual and linguistic information can yield improvements in visual question answering. The challenge we consider is to answer natural-questions related to a given image. The more general question we address in the context of this problem is how to encode visual and verbal information in a neural architecture. How to best do that is still unclear.  How are text entities bound to objects seen in images?  Are text and image best integrated late, allowing for independent analysis (\emph{late fusion}), or should the processing of one be conditioned on the analysis of the other (\emph{early fusion})?  How is cross-modal co-reference best encoded at all?  Does it make sense to ground words in the visual world before encoding sentence semantics?

In this work we gather evidence to answer these questions by designing the Bounding  Boxes  in  Text  Transformer, B2T2 for short, a neural architecture for multimodal encoding of natural language and images, and we evaluate B2T2 on the Visual Commonsense Reasoning benchmark (VCR, \citealt{zellers2019}).

Figure \ref{fig:rationale-needed} shows an illustrative example from the VCR benchmark.
VCR is well suited to test rich multimodal representations because it requires the analysis of images depicting people engaged in complex activities; it presents questions, answers and rationales created by human annotators rather than automatic generation; it has a clean multiple-choice interface for evaluation; and yet it is still challenging thanks to a careful selection of answer choices through adversarial matching. VCR has much longer questions and answers compared to other popular Visual Question Answering (VQA) datasets, such as VQA v1 \citep{VQA}, VQA v2 \citep{balanced_vqa_v2} and GQA \citep{hudson2019gqa}, requiring more modeling capacity for language understanding.

In our experiments, we found that early fusion of co-references between textual tokens and visual features of objects was the most critical factor in obtaining improvements on VCR. We found that the more visual object features we included in the model's input, the better the model performed, even if they were not explicitly co-referent to the text, and that positional features of objects in the image were also helpful. We finally discovered that our models for VCR could be trained much more reliably when they were initialized from pretraining on Conceptual Captions \citep{sharma2018conceptual}, a public dataset of about 3M images with captions. From the combination of these modeling improvements, we obtained a new model for visual question answering that achieves state-of-the-art on VCR, reducing  error rates by more than 25\% relative to the best published and documented model \citep{zellers2019}.

\section{Problem Formulation}

\begin{table}
\begin{scriptsize}
{\setlength{\tabcolsep}{2pt}
\begin{tabular}{ccp{4cm}} \\
Symbol & Type & Description \\ \midrule

$m$ & $\mathbb{N}$ & number of extracted bounding boxes \\

$n$ & $\mathbb{N}$ & number of tokens input to BERT \\

$k$ & $\mathbb{N}$ & number of positional embeddings for image coordinates, usually 56 \\

$d$ & $\mathbb{N}$ & visual features dimension, usually 2048 \\

$h$ & $\mathbb{N}$ & hidden dimension of BERT, usually 1024 \\

$l$ & $\{0, 1\}$ & a binary label \\

$I$ & $\mathbb{R}^{\cdot \times \cdot \times 3}$ & an image \\

$B$ & $\mathbb{R}^{m \times 4}$ & rectangular bounding boxes on $I$, as coordinates of opposite corners \\

$R$ & $\{0, 1\}^{m \times n}$ & matrix encoding which bounding boxes in $B$ correspond to which tokens in $T$ \\

$T$ & $\mathbb{N}^{n \times 2}$ & input tokens, each expressed as word piece id and token type \\

$\Phi$ & $\mathbb{R}^{\cdot \times \cdot \times 3} \rightarrow \mathbb{R}^d$ & a function to extract visual feature vectors from an image \\

$\pi$ & $\mathbb{R}^4 \rightarrow \mathbb{R}^d$ & a function to embed the position and shape of a bounding box \\

$\Psi$ & 
$\mathbb{R}^{n \times h} \rightarrow \mathbb{R}^{h}$ & a function to compute a passage embedding from per-token embeddings \\

$E$ & $\mathbb{N}^{n \times 2} \rightarrow \mathbb{R}^{n \times h}$ & non-contextualized token embeddings, encoding word piece ids, token types and positions \\ \bottomrule
\end{tabular}}
\end{scriptsize}
\caption{Glossary of mathematical symbols used in this work.}
\label{tab:glossary}
\end{table}

In this work, we assume data comprised of 4-tuples $(I, B, T, l)$ where
\begin{enumerate}
    \item $I$ is an image,
    \item $B = [b_1, \ldots, b_m]$ is a list of bounding boxes referring to regions of $I$, where each $b_i$ is identified by the lower left corner, height and width,
    \item $T = [t_1, \ldots, t_n]$ is a passage of tokenized text, with the peculiarity that some of the tokens are not natural language, but explicit references to elements of $B$, and
    \item $l$ is a binary label in $\{0, 1\}$.
\end{enumerate}

While it might seem surprising to mix natural text with explicit references to bounding boxes, this is actually a quite natural way for people to discuss objects in images and the VCR dataset is annotated in exactly this way.

We assume an image representation function $\Phi$ that converts an image, perhaps after resizing and padding, to a fixed size vector representation of dimension $d$.

We similarly assume a pretrained textual representation capable of converting any tokenized passage of text, perhaps after truncating or padding, into a vector representation of dimension $h$. We assume a context independent token representation $E$ in the shape of a vector of dimension $h$ for each token and a passage level representation $\Psi$ which operates on $E(T)$ and returns a passage level vector representation of dimension $h$.

We refer the reader to Table \ref{tab:glossary} for an overview of the notation used in this work. Full details on how the VCR dataset is encoded into this formalism are given in Section \ref{sec:data}.

\section{Models and Methods}

\begin{figure*}[t
\begin{minipage}{1.0\columnwidth}
\includegraphics[width=\columnwidth]{dual-encoder.pdf}
\caption{Dual Encoder architecture with late fusion. The model extracts a single visual feature vector from the entire image. Bounding boxes are ignored.
}
\label{fig:dual-encoder}
\end{minipage}\hspace{3\tabcolsep}%
\begin{minipage}{1.0\columnwidth}
\includegraphics[width=\columnwidth]{deep-bottom-up.pdf}
\caption{B2T2 architecture with early fusion. Bounding boxes are inserted where they are mentioned in the text and at the end of the input, as described in Sec.\ \ref{sec:data}.
}
\label{fig:deep-bottom-up}
\end{minipage}
\end{figure*}

We evaluate two main architectures: ``Dual Encoder'', a late fusion architecture where image and text are encoded separately and answer scores are computed as an inner product, and the full B2T2 model, an early fusion architecture where visual features are embedded on the same level as input word tokens.  Section~\ref{sec:ablations} will summarize experiments with model variants to answer the research questions laid out in the introduction and to analyze what works, and why.

\subsection{Dual Encoder}

Dual Encoders, discussed for example by \citet{wu2018starspace} and \citet{gillick2018end}, are models that embed objects of potentially different types into a common representation space where a similarity function can be expressed e.g.\ as a dot product or a cosine similarity. A notable example of a dual encoder for image classification is WSABIE, proposed by \citet{weston2011wsabie}.

Our Dual Encoder architecture is shown in Figure \ref{fig:dual-encoder}. We model the class distribution as
\[
   p(l = 1 | I, T) = \frac 1 {1 + e^{-\Psi(E(T))^\top D \Phi(I)}}
\]
where $D$ is a learned matrix of size $d \times h$. In this model, co-reference information is completely ignored, and the model must rely on fixed dimensional vectors for the late fusion of textual and visual contexts. However, we found this to be surprisingly competitive on VCR compared to published baselines, perhaps due to our choice of powerful pretrained models.


\subsection{B2T2}

Our B2T2 architecture is shown in Figure \ref{fig:deep-bottom-up}. 
We model the class distribution as
\[
  p(l|I, B, R, T) = \frac {e^{\Psi(E'(I, B, R, T)) \cdot a_l + b_l}}
         {\sum_{l'} e^{\Psi(E'(I, B, R, T)) \cdot a_{l'} + b_{l'}}}
\]
where $a_l \in \mathbb{R}^h$ and $b_l \in \mathbb{R}$ for $l \in \{0, 1\}$ are learned parameters. $E'(I, B, R, T)$ is a non-contextualized representation for each token and of its position in text, but also of the content and position of the bounding boxes. The key difference from ``Dual Encoder'' is that text, image and bounding boxes are combined at the level of the non-contextualized token representations rather than right before the classification decision.

The computation of $E'(I, B, R, T)$ is depicted in Figure \ref{fig:embedding}. More formally, for a given example, let matrix $R \in \{0, 1\}^{m \times n}$ encode the references between the bounding boxes in $B$ and the tokens in $T$, so that $R_{ij}$ is $1$ if and only if bounding box $i$ is referenced by token $j$. Then
\begin{align*}
& E'(I, B, R, T) = \\
& \;\; E(T) + \sum_{i=1}^m 
  R_i
   \left[ M (\Phi(\mbox{crop}(I, b_i)) + \pi(b_i))
   \right]^\top
\end{align*}
where $M$ is a learned $h \times d$ matrix, $\Phi(\mbox{crop}(I, b_i))$ denotes cropping image $I$ to bounding box $b_i$ and then extracting a visual feature vector of size $d$, and $\pi(b_i)$ denotes the embedding of $b_i$'s shape and position information in a vector of size $d$.

To embed the position and size of a bounding box $b$, we introduce two new learnable embedding matrices $X$ and $Y$ of dimension $k \times \frac d 4$. Let the coordinates of the opposite corners of $b$ be $(x_1, y_1)$ and $(x_2, y_2)$, after normalizing so that a bounding box covering the entire image would have $x_1 = y_1 = 0$ and $x_2 = y_2 = k$. Position embeddings are thus defined to be
\[
  \pi(b) = \mbox{concat}(X_{\lfloor x_1 \rfloor}, Y_{\lfloor y_1 \rfloor},
                         X_{\lfloor x_2 \rfloor}, Y_{\lfloor y_2 \rfloor})
\]

\subsection{Loss}
\label{sec:loss}

All of our models are trained with binary cross entropy loss using label $l$. Denoting $p := p(l=1|I,B,R,T)$, we have for each example
\begin{equation*}
\mathcal{L_{\text{BCE}}} = l \log p + (1 - l) \log (1 - p)
\end{equation*}

\subsection{Pretraining on Conceptual Captions}

Before training on VCR, we pretrain B2T2 on image and caption pairs using a Mask-LM pretraining technique like the one used in BERT \citep{devlin2018bert}. The setup used during pretraining is shown in Figure \ref{fig:mask-lm-pretraining}, where the model uses the image as additional context when filling in the mask.

We use two tasks for pretraining: (1) impostor identification and (2) masked language model prediction.
For the impostor task, we sample a random negative caption for each image and ask the model to predict whether the caption is correctly associated.
For mask-LM, we randomly replace tokens in the caption with the \texttt{[MASK]} token, and the model must predict the original token (see \citet{devlin2018bert} for more details).

Formally, the pretraining data consist of images $I$ and captions $T$. We do not consider bounding boxes during pretraining, so $B = \emptyset$. The binary label $l$ indicates whether the caption is an impostor or not.
The loss for impostor identification is binary cross entropy $\mathcal{L}_{\text{BCE}}$ with label $l$ as in~\ref{sec:loss}. 
We denote the loss for mask-LM as $\mathcal{L}_{\text{MLM}}$, which is the summed cross entropy of the predicted token distributions against the true tokens.

To ensure that our model correctly grounds the language to the image with the mask LM loss, we only use it for positive captions, zeroing it out for negative captions.
Our final objective is the sum of the losses:
\[ \mathcal{L} = \mathcal{L}_{\text{BCE}} + I[l = 1] \cdot \mathcal{L}_{\text{MLM}} \]
where $I[l=1]$ is an indicator for the label $l$ being positive for the image and caption pair.

We pretrain on Conceptual Captions \citep{sharma2018conceptual}, a dataset with over 3M images paired with captions.\footnote{We also tried pretraining on MS-COCO images and captions \citep{lin2014microsoft}, but found this to be ineffective. This could be because MS-COCO is smaller (with around 80k images, 400k captions).
}
We found empirically that pretraining improves our model slightly on VCR, but more importantly, allows our model to train stably. Without pretraining, results on VCR exhibit much higher variance. We refer the reader to Section \ref{sec:ablations} for an ablation analysis on the effect of pretraining.


\begin{figure}[t]
\includegraphics[width=\columnwidth,trim={0 2cm 0 2cm},clip]{embedding.pdf}
\caption{How input embeddings are computed in our B2T2 architecture.}
\label{fig:embedding}
\end{figure}


\begin{figure}[t]
\includegraphics[width=\columnwidth]{mask-lm-pretraining.pdf}
\caption{Mask-LM pretraining for B2T2.}
\label{fig:mask-lm-pretraining}
\end{figure}

\subsection{Implementation Details}
\label{sec:implementation}

We use ResNet-152\footnote{Publicly available at \url{tfhub.dev}} \cite{he2016identity} pretrained on ImageNet for $\Phi$, which yields a vector representation of size $d=2048$.  BERT-Large \citep{devlin2018bert} provides both $E$ and $\Psi$. The latter is a pretrained Transformer with 24 layers, 16 attention heads, and hidden size 1024. For BERT, $E$ corresponds to its token embeddings, $\Psi$ to the \texttt{[CLS]} token representation in the final layer, and so $\Psi(E(T))$ corresponds to the BERT passage representation of size $h=1024$.

We found empirically that it was slightly better to keep $\Phi$ fixed rather than fine-tuning it, but that it was of critical importance to fine-tune $\Psi$ and $E$ for the new task.

In all of our finetuning experiments we use the Adam optimizer \citep{kingma2014adam} and trained our models with a grid of hyperparameters: a learning rate of $2 \cdot 10^{-5}$ and $3 \cdot 10^{-5}$, for 3, 4, and 5 epochs with a linear learning rate decay, and two random seed for initialization.

To maximize performance on VCR, we also evaluate an ensemble of B2T2 models. Our ensemble is comprised of 5 identical B2T2 models, trained for 3 epochs with an initial learning rate of $2\cdot 10^{-5}$, but initialized with 5 different random seeds. The resulting class logits are then summed to obtain the ensemble scores.



\section{Data}
\label{sec:data}

\emph{Visual Commonsense Reasoning} (VCR, \url{visualcommonsense.com}, \citealt{zellers2019}) is a corpus that contains a sample of stills from movies. Questions and answers revolve around conclusions or assumptions that require knowledge external to the images. The associated task is to not only select a correct answer but also provide reasoning in line with common sense.
Matching our problem formulation given before, a VCR sample is defined as a tuple $(I, O, Q, A, R)$.
Here, $I$ is the image, and $O$ is a sequence of objects identified in the image.
A question $Q = [q_0, \ldots, q_k]$ is given, where tokens are either textual words or deictic references to objects in $O$. 
Each question contains a set of four answers $A = \{A_1, A_2, A_3, A_4\}$, with exactly one correct answer $A^*$.
Each response follows the schema of the queries. Finally, there is a set of four rationales $R = \{R_1, R_2, R_3, R_4\}$, with exactly one rationale $R^*$ identified as correct in supporting $A^*$. 

Each of the objects in $O = [(b_1, l_1), \ldots, (b_{|O|}, l_{|O|})]$ is identified in the image $I$ by bounding boxes $b_i$.  The objects are also labeled with their classes with a text token $l_i$.

The $Q\rightarrow A$ task is to choose $A^*$ given $(I, O, Q, A)$.   The $QA\rightarrow R$ task is to choose $R^*$ given $(I, O, Q, A^*, R)$. Finally, the $Q \to AR$ task is a pipeline of the two, where a model must first correctly choose $A^*$ from $A$, then correctly choose $R^*$ given $A^*$.


We adapt VCR to our problem formulation by converting each VCR example to four instances for the $Q \rightarrow A$ task, one per answer in $A$, and four instances for the $QA \rightarrow R$ task, one per rationale in $R$. We construct the text for the instances in the $Q \rightarrow A$ task as
\begin{align*}
  [&\mbox{[CLS]}, [b_0], q_0, \ldots, \mbox{[SEP]}, \\
   &a_0, \ldots, \mbox{[SEP]}, l_1, [b_1], \ldots, l_p, [b_p]]
\end{align*}
and in the $QA \rightarrow R$ task as
\begin{align*}
  [&\mbox{[CLS]}, [b_0], q_0, \ldots, \mbox{[SEP]}, a^*_0, \ldots, \\
  & r_0, \ldots, \mbox{[SEP]}, l_1, [b_1], \ldots, l_p, [b_p]].
\end{align*}
where $\mbox{[CLS]}, \mbox{[SEP]}$ are special tokens for BERT.

Here, $[b_0]$ is a bounding box referring to the entire input image. $q_0, \ldots$ are all question tokens, $a_0, \ldots$ answer tokens, $a^*_0, \ldots$ answer tokens for the correct answer, and $r_0, \ldots$ rationale tokens.
We append the first $p$ bounding boxes in $O$ with class labels to the end of the sequence (in our experiments, we use $p = 8$), and for objects referenced in $Q, A, R$, we prepend the class label token (i.e. $[b_i]$ becomes $l_i, [b_i]$).
We assign the binary label $l$ to every instance to represent whether the answer or rationale choice is the correct one.




\section{Experimental Results}

\subsection{VCR Task Performance}

Our final results on the VCR task are shown in Table \ref{tab:results}. Our Dual Encoder model worked surprisingly well compared to \citet{zellers2019}, surpassing the baseline without making use of bounding boxes.
We also evaluate a Text-Only baseline, which is similar to the Dual Encoder model but ignores the image.
The ensemble of B2T2 models, pretrained on Conceptual Captions, obtained absolute accuracy improvements of $8.9\%$, $9.8\%$ and $13.1\%$ compared to the published R2C baseline for the $Q \rightarrow A$, $QA \rightarrow R$, and $Q \rightarrow AR$ tasks respectively. At the time of this writing (May 22, 2019), both our single B2T2 and ensemble B2T2 models outperform all other systems in the VCR leaderboard.


\input{results-table}

\subsection{Ablations}
\label{sec:ablations}
\begin{table}[t]
    \small
    \centering
    \begin{tabular}{c|c}
         & $Q\to A$  \\
        \toprule
        Dual Encoder & 66.8 \\
        \midrule
        No bboxes & 67.5 \\
        Late fusion & 68.6 \\
        BERT-Base & 69.0 \\ 
        ResNet-50 & 70.4 \\
        No bbox class labels & 70.9 \\
        Fewer appended bboxes ($p=4$) & 71.0 \\
        No bbox position embeddings & 71.6 \\
        \midrule
        Full B2T2 & 71.9 \\
        \bottomrule
       
    \end{tabular}
    \caption{Ablations for B2T2 on VCR dev. The Dual Encoder and the full B2T2 models are the main models discussed in this work. All other models represent ablations from the full B2T2 model.}
    \label{tab:ablations}
\end{table}

To better understand the reason for our improvements, we performed a number of ablation studies on our results, summarized in Table \ref{tab:ablations}. We consider ablations in order of decreasing impact on the VCR dev set $Q \rightarrow A$ accuracy.

\begin{figure}
    \centering
    \includegraphics[width=\columnwidth]{pretraining_effect.pdf}
    \caption{Boxplot of dev $Q \rightarrow A$ accuracy on VCR with and without pretraining. Pretraining on Conceptual Captions lowers variance when fine-tuning on VCR, from a grid search on multiple random seeds, learning rates, and VCR training epochs. }
    \label{fig:pretraining}
\end{figure}

{\bf Use of Bounding Boxes.} The bounding boxes considered by our model turns out to be the most important factor in improving the accuracy of our model. Without any bounding boxes we obtain $67.5\%$ accuracy, just above the accuracy of the dual encoder. With 4 instead of 8 appended bounding boxes we obtain $71\%$ accuracy. With 8 bounding boxes, but no textual labels from the bounding boxes in the text we obtain $70.9\%$ accuracy, showing that our model can make use of labels for detected objects. Example 1 in Table \ref{tab:examples} shows an example that our models can only get right if bounding box 5 is available.

{\bf Late Fusion vs.\ Early Fusion.} The second most important architectural choice in our model is to combine visual information at the level of context independent token embeddings, rather than at the highest levels of the neural representation. If in the the full B2T2 model we add visual embeddings in the last layer of BERT rather than in the first, we lose $3.3\%$ accuracy.

{\bf Effect of Textual Model Size.} The original VCR work by \citet{zellers2019} made use of BERT-base, while we use BERT-large to initialize our models. To test how much of our improvements are simply due to our model being larger, we retrained B2T2 models using BERT-base and found that we lose 2.9\% accuracy.

{\bf Effect of Visual Model Size.} How important is the choice of the visual model in the performance of B2T2? As further discussed in the error analysis section of this work, we suspect that B2T2 could be significantly improved by extending the visual features to represent more than just objects, but also activities, expressions and more. However it appears that even the size of the object detection model is important. If we swap out ResNet-152 for ResNet-50, accuracy decreases by $1.5\%$.

{\bf Pretraining.} We found that performance improvements from pretraining are quite small, around 0.4\% accuracy, but initializing from a pretrained model heavily reduces variance of results. We show this effect in Figure~\ref{fig:pretraining} over the grid of learning rates, random seeds, and training epochs described in Section \ref{sec:implementation}.

{\bf Position of Bounding Boxes} We additionally investigated the effect of removing position information from the model. The benefit of having bounding box positional embeddings is the smallest of the ones we considered. A model trained without positional embeddings only loses 0.3\% accuracy compared to the full model.

\subsection{Error Analysis}

\input{error-analysis}

\section{Related Work}

Modeling visual contexts can aid in learning useful sentence representations \citep{kiela2017learning} and even in training language models \citep{ororbia2019acl}. This paper takes these more general ideas to a downstream task that requires modeling of visual input.  Similar to B2T2, VideoBERT \citep{sun2019videobert} jointly processes video frames and text tokens with a Transformer architecture. However, VideoBERT cannot answer questions, nor does the model consider bounding boxes.

Our B2T2 model is similar to the Bottom-Up Top-Down attention model \citep{anderson2018bottom} in how bounding boxes generated at preprocessing time are attended to by the VQA model. ``Bottom-Up'' refers to the idea of attending from the text to the bounding boxes of objects detected in the image, while ``Top-Down'' refers to the idea of attending to regions constructed as a regular grid over the image. The Bottom-Up Top-Down model however reduces the text to a fixed length vector representation before attending to image regions, while B2T2 instead treats image regions as special visual tokens mixed in the text. In this sense, Bottom-Up Top-Down model is a late fusion model, while B2T2 is early fusion.

The Neuro-Symbolic Concept Learner \citep{mao2019neuro} also uses bounding boxes to learn visually grounded concepts through language. The Neuro-Symbolic Concept Learner however relies on a semantic parser to intepret language, while B2T2 
uses a Transformer to construct a joint representation of textual tokens and visual tokens.

Another recently proposed model for VQA is MAC \citep{hudson2018compositional}. As presented, MAC does not make use of bounding boxes, which makes it a Top-Down model in the nomenclature of \citet{anderson2018bottom}. MAC also reduces the textual information to a vector of fixed length. However MAC makes use of a new neural architecture designed to perform an explicit multi-step reasoning process and is reported to perform better than \citet{anderson2018bottom} on the GQA dataset \citep{hudson2019gqa}.

After the submission of this paper, several new works were published with excellent results on VCR, in some cases exceeding the performance of our system. In particular we mention ViLBERT \citep{lu2019vilbert}, VL-BERT \citep{su2019vl}, Unicoder-VL \citep{li2019unicoder}, and VisualBERT \citep{li2019visualbert}.

VCR is only one of several recent datasets pertaining to the visual question answering task. VQA \citep{VQA, balanced_binary_vqa, balanced_vqa_v2} contains photos and abstract scenes with questions and several ground-truth answers for each, but the questions are less complex than VCR's. CLEVR \citep{johnson2017clevr} is a visual QA task with compositional language, but the scenes and language are synthetic. GQA \citep{hudson2019gqa} uses real scenes from Visual Genome, but the language is artificially generated.
Because VCR has more complex natural language than other datasets, we consider it the best evaluation of a model like B2T2, which has a powerful language understanding component. 

\section{Conclusion}

In this work we contrast different ways of combining text and images when powerful text and vision models are available. We picked BERT-Large \cite{devlin2018bert} as our text model, ResNet-152 \cite{he2016identity} as our vision model, and the VCR dataset \cite{zellers2019} as our main benchmark.

The early-fusion B2T2 model, which encodes sentences along with links to bounding boxes around identified objects in the images, produces the best available results in the visual question answering tasks.  A control model, implementing late fusion (but the same otherwise), performs substantively worse.  Thus, grounding words in the visual context should be done early rather than late.

We also demonstrate competitive results with a Dual Encoder model, matching state-of-the-art on the VCR dataset even when textual references to image bounding boxes are ignored.  We then showed that our Dual Encoder model can be substantially improved by deeply incorporating in the textual embeddings visual features extracted from the entire image and from bounding boxes.  We finally show that pretraining our deep model on Conceptual Captions with a Mask-LM loss yields a small additional improvement as well as much more stable fine-tuning results.


