\section{Introduction}
\input{sections/main/introduction.tex}

\section{Related Work}
\input{sections/main/related_work.tex}

\section{Approach}
\input{sections/main/approach.tex}

\section{Experiments}
\input{sections/main/experiments.tex}

\section{Conclusions}
\input{sections/main/conclusions.tex}
\newpage


\subsection{Neural Modules for Visual Question Answering}
\label{sec:appraoch:modules:vqa}

The main technical foundation of our model is the neural module network (NMN) \cite{andreas16cvpr}.
In this section, we briefly recap NMNs and more specifically, the attentional
modules from \cite{hu2017learning}.
In the next section, we discuss novel modules we propose to handle additional 
challenges in visual dialog.

For a module $m$, let $x_{vis}$ and $x_{txt}$ be the input image and text 
embeddings, respectively.
In particular, the image embeddings $x_{vis}$ are spatial activation
maps of the image $I$ from a convolutional neural network.
The text embedding $x_{txt}$ is computed as a weighted sum of embeddings
of words in the question $Q_t$ using the soft attention weights $\alpha$
predicted by a program generator for module $m$ (more details in 
Sec\onedot\ref{sec:prog_predict}).
Further, let $\{a_i\}$ be the set of $n_m$ single-channel spatial maps 
corresponding to the spatial image embeddings, where $n_m$ is the number of
attention inputs to $m$.
Denoting the module parameters with $\theta_m$, a neural module $m$ is 
essentially a parametric function 
$y = f_m(x_{vis}, x_{txt}, \{a_i\}_{i=1}^{n_m}; \theta_m)$.
The output from the module $y$ can either be a spatial image attention map 
(denoted by $a$) or a context vector (denoted by $c$), depending on the module.
The output spatial attention map $a$ feeds into next level modules while a context 
vector $c$ is used to obtain the final answer $A_t$.
The upper part of \Table\ref{tab:module_table} lists modules we adopt from 
prior work, with their functional forms.
We shortly summarize their behavior.
A \texttt{Find} module localizes objects or attributes by producing an 
attention over the image.
The \texttt{Relocate} module takes in an input image attention and performs
necessary spatial relocations to handle relationships like \textit{`next to'},
\textit{`in front of'}, \textit{`beside'}, etc\onedot} \defvs\onedot{vs\onedot.
Intersection or union of attention maps can be obtained using \texttt{And} and \texttt{Or}, respectively.
Finally, \texttt{Describe}, \texttt{Exist}, and \texttt{Count} input an 
attention map to produce the context vector by describing an attribute, 
checking for existence, or counting, respectively, in the given input attention
map.
As noted in \cite{hu2017learning}, these modules are designed and named for
a potential `atomic' functionality.
However, we do not enforce this explicitly and let the modules discover their
expected behavior by training in an end-to-end manner.


\input{sections/main/module_table.tex}
\subsection{Neural Modules for Coreference Resolution}
\label{sec:appraoch:modules:coref}
We now introduce novel components and modules to handle visual dialog.

\myparagraph{Reference Pool ($P_{ref}$).}
The role of the reference pool is to keep track of entities seen so far
in the dialog.
Thus, we design $P_{ref}$ to be a dictionary of key-value pairs 
$(x_{txt}, a)$ 
for all the \texttt{Find} modules instantiated while answering
previous questions $(Q_i)_{i=1}^{t-1}$.
Recall that \texttt{Find} localizes objects/attributes specified by 
$x_{txt}$, and thus by storing each output attention map $y$, we now have 
access 
to all the entities mentioned so far in the dialog with their corresponding
visual groundings.
Interestingly, even though $x_{txt}$ and $y$ are intermediate 
outputs from our model, both are easily interpretable, making our reference
pool a \emph{semantic dictionary}.
To the best of our knowledge, our model is the first to attempt explicit,
interpretable coreference resolution in visual dialog.
While \cite{paul2017visual} maintains a dictionary similar to $P_{ref}$,
they do not consider word/entity level coreferences nor do their keys lend 
similar interpretability as ours.
With $P_{ref} = \{(x_p^{(i)}, a_p^{(i)})\}_i$ as input to \texttt{Refer},
we can now resolve references in $Q_t$.

\myparagraph{\texttt{Refer} Module.}
This novel module is responsible for 
resolving references in the question $Q_t$ and ground them in the conversation 
history $H$.
To enable grounding in dialog history, we generalize the above formulation
 to give the module access to a pool of references $P_{ref}$ of previously 
 identified entities.
Specifically, \texttt{Refer} only takes the text embedding $x_{txt}$ and the 
reference pool $P_{ref}$ as inputs, and resolves the entity represented by 
$x_{txt}$ in the form of a soft attention $\alpha$ over $Q_t$.
in this section after introducing $P_{ref}$.
For the example shown in \Fig\ref{fig:model_fig},
$\alpha$ for \texttt{Refer} attends to \textit{`it'}, indicating the phrase it
is trying to resolve.

At a high level, \texttt{Refer} treats $x_{txt}$ as a `query' and retrieves
the most likely match from $P_{ref}$ as measured by some similarity with
respect to keys $\{x_p^{(i)}\}_i$ in $P_{ref}$.
The associated image attention map of the best match is used as the visual 
grounding for the phrase that needed resolution (i.e\onedot} \def\Ie{I.e\onedot \textit{`it'}).
More concretely, we first learn a \textit{scoring network} which when given a 
query $x_{txt}$ and a possible candidate $x_p^{(i)}$, returns a scalar value 
$s_i$ indicating how likely these text features refer to the same entity
\eqref{eq:refer-score-1}.
To enable \texttt{Refer} to consider the sequential nature of dialog when 
assessing a potential candidate, we additionally provide $\Delta_i t$, a measure of the `distance' of a candidate $x_p^{(i)}$ from $x_{txt}$ 
in the dialog history, as input to the scoring network.
$\Delta_i t$ is formulated as the absolute difference between the round of
$x_{txt}$ (current round $t$) and the round when $x_p^{(i)}$ was first
mentioned.
Collecting these scores from all the candidates, we apply a softmax function
to compute contributions $\tilde{s}_i$ from each entity in the pool 
\eqref{eq:refer-score-2}.
Finally, we weigh the corresponding attention maps via these contributions
to obtain the visual grounding $a_{out}$ for $x_{txt}$
\eqref{eq:refer-score-3}.
\begin{multicols}{2}
\vspace*{-30pt}
\begin{align}
s_i &= \text{MLP}([x_{txt}, x_p^{(i)}, \Delta_i t]) \label{eq:refer-score-1}\\
\tilde{s}_i &= \text{Softmax}(s_i) \label{eq:refer-score-2}
\end{align}
\columnbreak
\begin{align}
a_{out} &= \sum_{i=1}^{|P_{ref}|} \tilde{s}_i a_p^{(i)} \label{eq:refer-score-3}
\end{align}
\end{multicols}
\vspace*{-20pt}

\myparagraph{\texttt{Not} Module.}
Designed to focus on regions of the image \textbf{not} attended by the input 
attention map $a$, it outputs $y = \text{norm}_{L_1}(1-a)$, where 
norm$_{L_1}(.)$ normalizes the entries to sum to one.
This module is used in \texttt{Exclude}, described next.

\myparagraph{\texttt{Exclude} Module.}
To handle questions like \textit{`What other red things are present?'}, which 
seek other objects/attributes in the image than those specified by an input
attention map $a$, we introduce yet another novel module -- \texttt{Exclude}.
It is constructed using \texttt{Find}, \texttt{Not}, and \texttt{And} modules
as $y =$ \texttt{And}[\texttt{Find}[$x_{txt}, x_{vis}$], \texttt{Not}[$a$]],
where $x_{txt}$ is the text feature input to the \texttt{Exclude} module, for 
example, \textit{`red things'}.
More explicitly, \texttt{Find} first localizes all objects instances/attributes 
in the image.
Next, we focus on regions of the image other than those specified by $a$ using 
\texttt{Not}[$a$].
Finally, the above two outputs are combined via \texttt{And} to obtain the 
output $y$ of the \texttt{Exclude} module.


\subsection{Program Generation}
\label{sec:prog_predict}
A \textit{program} specifies the network layout for the neural modules 
for a given question $Q_t$.
Following \cite{hu2017learning}, it is serialized through the reverse 
polish notation (RPN) \cite{10.2307/2001990}.
This serialization helps us convert a hard, structured prediction problem into
a more tractable sequence prediction problem.
In other words, we need a program predictor to output a series of module
tokens in order, such that a valid layout can be retrieved from it.
There are two primary design considerations for our predictor.
First, in addition to the program, our predictor must also output a
soft attention $\alpha_{ti}$, over the question $Q_t$, for every module 
$m_i$ in the program.
This attention is responsible for the \emph{correct} module instantiation
in the current context.
For example, to answer the question \textit{`What color is the cat sitting next
to the dog?'}, a \texttt{Find} module instance attending to \textit{`cat'}
qualitatively serves a different purpose than one attending to \textit{`dog'}.
This is implemented by using the attention over $Q_t$ to compute the text 
embedding $x_{txt}$ that is directly fed as an input to the module during 
execution.
Second, to decide whether an entity in $Q_t$ has been seen before in the
conversation, it must be able to `peek' into the history $H$.
Note that this is unique to our current problem and does not exist in \cite{hu2017learning}.
To this effect, we propose a novel augmentation of attentional recurrent neural 
networks~\cite{bahdanau_iclr15} with memory \cite{DBLP:journals/corr/WestonCB14}
to address both the requirements (\Fig\ref{fig:model_fig}).

The program generation proceeds as follows.
First, each of the words in $Q_t$ are embedded to give $\{w_{ti}\}_{i=1}^T$,
where $T$ denotes the number of tokens in $Q_t$.
We then use a \textit{question encoder}, a multi-layer LSTM, to process
$w_{ti}$'s, resulting in a sequence of hidden states $\{\hat{w}_{ti}\}_{i=1}^T$
\eqref{eq:ques-enc-1}.
Notice that the last hidden state $h_T$ is the question encoding, which we
denote with $q_t$.
Next, each piece of history $(H_i)_{i=0}^{t-1}$ is processed in a similar
way by a \textit{history encoder}, which is a multi-layer LSTM akin to the
question encoder.
This produces encodings $(h_i)_{i=0}^{t-1}$ \eqref{eq:hist-mem-1} that serve 
as memory units to help the program predictor `peek' into the conversation
history.
Using the question encoding $q_t$, we attend over the history encodings 
$(h_i)_{i=0}^{t-1}$, and obtain the history vector $\hat{h}_t$ 
\eqref{eq:hist-mem-2}.
The history-agnostic question encoding $q_t$ is then fused with the history 
vector $\hat{h}_t$ via a fully connected layer to give a history-aware
question encoding $\hat{q}_t$ \eqref{eq:prog-dec-1}, which is fed into the \textit{program decoder}.

\input{sections/main/prog_generator_eqs.tex}

The decoder is another multi-layer LSTM network (with hidden states 
$\{d_{ti}\}$) which, at every time step $i$,
produces a soft attention map $\alpha_{ti}$ over the input sequence ($Q_t$) 
\cite{bahdanau_iclr15}.
This soft attention map for each module is used to compute the corresponding
text embedding, $x_{txt} = \sum_{j} \alpha_{ti}^{(j)} w_{tj}$.
Finally, to predict a module token $m_i$ at time step $i$, a weighted sum
of encoder hidden states $e_{ti}$ \eqref{eq:prog-dec-5} and the history-aware 
question vector $\hat{q}_t$ are combined via another fully-connected
layer \eqref{eq:prog-dec-6}, followed by a softmax to give a distribution 
$P(m_i | \{m_k\}_{k=1}^{i-1}, Q_t, H)$ over the module tokens 
\eqref{eq:prog-dec-7}.
During training, we minimize the cross-entropy loss 
$\mathcal{L}_Q^{prog}$ between this predicted distribution and the ground truth 
program tokens.
\Fig\ref{fig:model_fig} outlines the schematics of our program generator.


\myparagraph{Modules on captions.}
As the image caption $C$ is also a part of the dialog (history $H_0$ at round $0$),
it is desirable to track entities from $C$ via the coreference pool 
$P_{ref}$.
To this effect, we propose a novel extension of neural module networks to 
captions by using an auxiliary task that checks the alignment of a (caption, 
image) pair.
First, we learn to predict a program from $C$, different from those generated 
from $Q_t$, by minimizing the negative log-likelihood $\mathcal{L}_C^{prog}$,
akin to $\mathcal{L}_Q^{prog}$, of the ground truth caption program.
Next, we execute the caption program on two images $I^+ = I$ and $I^-$ 
(a random image from the dataset), to produce caption context vectors $c_C^+$
and $c_C^-$, respectively.
Note that $c_C^+$ and $c_C^-$ are different from the context vector $c_t$ 
produced from execution of the question program.
Finally, we learn a binary classifier on top to output classes $+1/-1$ for 
$c_C^+$ and $c_C^-$, respectively, by minimizing the binary cross entropy loss 
$\mathcal{L}_C^{aux}$.
The intuition behind the auxiliary task is:
to rightly classify aligned $(C, I^+)$ from misaligned $(C, I^-)$,
the modules will need to localize and focus on salient entities in the caption.
These entities (specifically, outputs from \texttt{Find} in the caption program)
are then collected in $P_{ref}$ for explicit coreference resolution on $Q_t$.


\myparagraph{Entities in answers.}
Using an analogous argument as above, answers from the previous rounds 
$\{A_i\}_{i=1}^{t-1}$ could have entities necessary to resolve coreferences in
$Q_t$.
For example,
\textit{`Q: What is the boy holding? A: A ball. Q: What color is it?'}
requires resolving `it' with the `ball' mentioned in the earlier answer.
To achieve this, at the end of round $t-1$, we encode 
$H_{t-1} = (Q_{t-1}, A_{t-1})$ as $h^{ref}_t$ using a multi-layer LSTM,
obtain the last image attention map $a$ fed to the last module in the program
that produced the context vector $c_t$, and add $(h^{ref}, a)$ as an additional 
candidate to the reference pool $P_{ref}$.
Notice that $h^{ref}$ contains the information about the answer $A_{t-1}$ in
the context of the question $Q_{t-1}$, while $a$ denotes the image attention
which was the last crucial step in arriving at $A_{t-1}$ in the earlier round.
In resolving coreferences in $Q_t$, if any, all the answers from previous
rounds now become potential candidates by virtue of being in $P_{ref}$.



\subsection{Other Model Components}
\label{sec:prog_execute}

\myparagraph{Program Execution.}
This component takes the generated program and associated text features 
$x_{txt}$ for each participating module, and executes it.
To do so, we first deserialize the given program from its RPN to a hierarchical
module layout.
Next, we arrange the modules dynamically according to the layout, giving us the 
network to answer $Q_t$.
At this point, the network is a simple feed-forward neural network, where we
start the computation from the leaf modules and feed outputs activations from 
modules at one layer as inputs into modules at the next layer 
(see \Fig\ref{fig:model_fig}).
Finally, we feed a context vector $c_t$ produced from the last module into
the next answer decoding component.

\myparagraph{Answer Decoding.}
\label{sec:ans_decode}
This is the last component of our model that uses the context vector $c_t$ to 
score answers from a pool of candidates $\mathcal{A}_t$, based on their
correctness.
The answer decoder: 
(a) encodes each candidate $A_t^{(i)}\in \mathcal{A}_t$ with a multi-layer LSTM
to obtain $o_t^{(i)}$,
(b) computes a score via a dot product with the context vector, i.e\onedot} \def\Ie{I.e\onedot, $c_t^T o_t^{(i)}$, and
(c) applies a softmax activation to get a distribution over the candidates.
During training, we minimize the negative log-likelihood $\mathcal{L}_A^{dec}$
of the ground truth answer $A^{gt}_t$.
At test time, the candidate with the maximum score is picked as $\mathcal{A}_t$.
Using nomenclature from \cite{visdial}, this is a \textit{discriminative} 
decoder.
Note that our approach is not limited to a discriminative decoder, but can also 
be used with a \textit{generative} decoder (see supplement).

\myparagraph{Training Details.}
Our model components have fully differentiable operations within them.
Thus, to train our model, we combine the supervised loss terms from both
program generation 
$\{\mathcal{L}_Q^{prog}, \mathcal{L}_C^{prog}, \mathcal{L}_C^{aux}\}$
 %
 %
and answer decoding $\{\mathcal{L}^{dec}_A\}$, and minimize the sum total loss 
$\mathcal{L}^{total}$.

\section*{Overview of Supplement}
The supplement is organized as follows:
\begin{itemize}
\setlength\itemsep{1em}
\item Sec\onedot\ref{sup:disc_results} shows the results of our model using a 
	discriminative decoder with image features extracted using ImageNet 
    pretrained ResNet-152 \cite{he16cvpr},
    showing superior performance of our 
	explicit coreference model CorefNMN\xspace,
\item Sec\onedot\ref{sup:gen_results} details our experiments with a generative 
	answer decoder,
\item Implementation details for our experiments are given in 
	Sec\onedot\ref{sup:implement_details}, and
\item Schematics of our novel \texttt{Refer} module are in 
	\Fig\ref{fig:refer_visual}, \Fig\ref{fig:nmn_caption_supp} visualizes 
    the auxiliary task used to run modules on captions, a novel way to handle 
    captions at a fine word-level granularity, and \Fig\ref{fig:visdial_qual_2}
    shows another qualitative example from VisDial.
\end{itemize}

\section{Discriminative Decoder Experiments}
\label{sup:disc_results}
\myparagraph{Comparisons with ResNet-152 features.}
As mentioned in Sec. 4.2 of the main paper, the models trained on
VisDial v0.9 used an ImageNet pretrained VGG \cite{simonyan_iclr15} to extract
the image features $x_{vis}$.
In this section, we present results where a pretrained ResNet-152 \cite{he16cvpr}
was used to obtain the image features for our CorefNMN\xspace model, in \reftab{tab:visdial_disc_result_table_resnet}.
For a fair comparison, we obtained performance metrics from the authors for few
of these baselines (MNQIH-G, LF-QH-G), and retrain NMN with ResNet-152 features.

\section{Generative Decoder Experiments}
\label{sup:gen_results}

The main paper describes a discriminative answer decoder (Sec.3.4) and presents results for the same (Sec. 4.2) on VisDial v0.9 dataset.
As a reminder, a discriminative decoder takes the context vector $c_t$ as an input, and scores candidate answers according to their correctness.
In other words, a discriminative decoder needs to be presented with a list of
candidate answers and cannot `generate' novel answers.
We now introduce a generative answer decoder and present results on VisDial 
v0.9 dataset.

\myparagraph{Generative Answer Decoder} is a language model composed of a 
multi-layer LSTM with $c_t$ as its initial state. During training, we minimize
the negative log-likelihood $\mathcal{L}^{dec}_A$ of the ground truth answer
$A^{gt}_t$ with respect to the model.
At test time, we use the decoder to score all candidates in the answer pool 
$\mathcal{A}_t$ by model log-likelihood, and rank them accordingly.
Note that this ranking of candidate answers is done to comply with the 
evaluation protocol of VisDial v0.9, and is not a limitation of generative
answer decoder, unlike the discriminative one.
Thus, the generative answer decoder can potentially be used to generate novel
answers to a given question in the visual dialog via language generation.

\myparagraph{Models and baselines.}
We denote our model, as described in Sec\onedot\ref{sec:approach}, with
CorefNMN\xspace to indicate that the model uses history $H$.
We also consider ablations which do not have the memory network augmented
program generator (CorefNMN\xspace{}$\backslash$Mem), or, the auxiliary loss 
$\mathcal{L}_C^{aux}$ to train modules on captions 
(CorefNMN\xspace{}$\backslash\mathcal{L}_C^{aux}$), and a combination of both
(CorefNMN\xspace{}$\backslash$Mem$\backslash\mathcal{L}_C^{aux}$).
As strong baselines,
we consider: 
(a) neural module network without history \cite{hu2017learning} with answer generation,
(b) the best \textit{generative} model based on memory networks MN-QIH-G from 
\cite{visdial}, in addition to their LF-QIG-G and HRE-QIH-G models, and
(c) history-conditioned image attentive encoder (HCIAE-G-MLE) \cite{lu2017best}.
We do not consider the HCIAE-G-DIS model with perceptual loss \cite{lu2017best}
as its contribution is complementary to our model.
Our model uses  ImageNet pretrained ResNet-152 
 \cite{he16cvpr} to extract features for images $x_{vis}$, while
some of these baseline models originally use VGG-16 \cite{simonyan_iclr15}
features.
For a fair comparison, we obtained performance metrics from the authors for few
of these baselines (MN-QIH-G, LF-QH-G), and report the others (HCIAE-G-MLE) as 
is.

\input{sections/main/visdial_disc_result_table_both.tex}



\myparagraph{Results.}
We summarize our observations from \Table\ref{tab:visdial_result_table} below:
(a) Our CorefNMN\xspace outperforms all other approaches according to R$@5$, R$@10$,
  and mean rank metrics, highlighting the importance of explicitly resolving
  coreferences for visual dialog.
  Specifically, our mean rank of $15.69$ is a $4\%$ improvement over the NMN 
  baseline.
(b) However, the NMN baseline has a higher R$@1$, and perhaps as a result,
	the best MRR.
    A possible reason could be due to the noisy supervision from the 
    out-of-domain, automatic, text-based coreference tool, as we entirely rely
    on it to predict \texttt{Refer}, the module responsible for coreference
    resolution.
(c) Our novel way of handling captions using modules (indicated
	$\mathcal{L}_C^{aux}$) boosts the full model, while the 
    hurting the ablation without the memory augmentation.
    That is, CorefNMN\xspace{}$\backslash$Mem$\backslash\mathcal{L}_C^{aux}$ is better 
    than CorefNMN\xspace{}$\backslash$Mem across all metrics, while 
    CorefNMN\xspace{}$\backslash \mathcal{L}_C^{aux}$ is uniformly worse than CorefNMN\xspace.

\input{sections/main/visdial_gen_result_table.tex}

\input{sections/supp/refer_visuals.tex}

\input{sections/supp/nmn_caption.tex}

\section{Implementation Details}
\label{sup:implement_details}
\input{sections/supp/implement_details.tex}

\begin{figure}
	\centering
    \includegraphics[width=0.98\textwidth]{figures/visdial_qualitative_ex2_v1.png}
    \caption{
    Qualitative example on VisDial dataset.}
    \label{fig:visdial_qual_2}
\end{figure}








\subsection{MNIST Dialog Dataset}

\noindent
\textbf{Dataset.}
The dialogs in the MNIST dialog dataset \cite{paul2017visual} are grounded in
images composed from a $4 \times 4$ grid of MNIST digits \cite{lecun-mnisthandwrittendigit-2010}.
Digits in the grid have four attributes---digit class ($0-9$), 
color, stroke, and background color.
Each dialog has $10$ question-answer pairs, where the questions are
generated through language templates, and the answers are single words.
Further, the questions are designed to query attributes of target digit(s),
count digits with similar attributes, etc\onedot} \defvs\onedot{vs\onedot, all of which need tracking of the
target digits(s) by resolving references across dialog rounds.
Thus, coreference resolution plays a crucial part in the reasoning required to
answer the question, making the MNIST dataset both interesting and challenging
(\Fig\ref{fig:mnist_qualitative}).
The dataset contains $30k$ training, $10k$ validation, and $10k$ test images,
with three $10$-round dialogs for each image.

\input{sections/main/mnist_result_table.tex}

\noindent
\textbf{Models and baselines.}
Taking advantage of single-word answers in this dataset, 
we simplify our answer decoder to be a $N$-way classifier, where $N$ is the number of possible answers.
Specifically, the context vector $c_t$ now passes through
a fully connected layer of size $N$, followed by softmax activations to give
us a distribution over possible answer classes.
At training time, we minimize the cross-entropy $\mathcal{L}_A^{dec}$ of the 
predicted answer distribution with the ground truth answer, at every round.
Note that single-word answers also simplify evaluation as answer accuracy can 
now be used to compare different models.
We further simplify our model by removing the memory augmentation to the
program generator, i.e\onedot} \def\Ie{I.e\onedot, $\hat{q}_t = q_t$ \eqref{eq:prog-dec-1}, and denote it
as CorefNMN\xspace.
In addition to the full model, we also evaluate an ablation, 
CorefNMN\xspace{}$\backslash$Seq, without $\Delta_i t$ that additionally captured 
sequential nature of dialog (see \texttt{Refer} description).
We compete against the explicit reasoning model (NMN) \cite{hu2017learning} and
a comprehensive set of baselines AMEM, image-only (I), and question-only (Q), 
all from \cite{paul2017visual}.

\myparagraph{Supervision.}
In addition to the ground truth answer, we also need program supervision for 
questions to learn the program generation.
For each of the $5$ `types' of  questions, we manually create one program which
we apply as supervision for all questions of the corresponding type.
The type of question is provided with the question.
Note that our model needs program supervision only while training, and uses
predictions from program generator at test time.


\myparagraph{Results.}
\Table\ref{tab:mnist_quantitative} shows the results on MNIST dataset.
The following are the key observations:
(a) The text-only Q ($36.6\%$) and image-only I ($20.2\%$) do not perform well,
perhaps as expected as MNIST Dialog needs resolving strong coreferences to 
arrive at the correct answer.
For the same reason, NMN \cite{hu2017learning} has a low accuracy of $23.8\%$.
Interestingly, Q outperforms NMN by around $13\%$ (both use question and image,
but not history), possibly due to the explicit reasoning nature of NMN 
prohibiting it from capturing the statistic dataset priors.
(b) Our CorefNMN\xspace outperforms all other models with near perfect accuracy of $99.3\%$.
Examining the failure cases reveals that most of the mistakes made by CorefNMN\xspace was
due to misclassifying qualitatively hard examples from the original MNIST 
dataset.
(c) Factoring the sequential nature of the dialog additionally in the model is
beneficial, as indicated by the $10.6\%$ improvement in CorefNMN\xspace, and $7.2\%$ in
AMEM.
Intuitively, phrases with multiple potential referents, more often
than not, refer to the most recent referent, as seen in \Fig\ref{fig:teaser}, 
where \textit{`it'} has to be resolved to the closest referent in history.
\Fig\ref{fig:mnist_qualitative} shows a qualitative example.

\input{sections/main/visdial_disc_result_table.tex}

\subsection{VisDial v0.9 Dataset}

\noindent
\textbf{Dataset.}
The VisDial dataset \cite{visdial} is a crowd-sourced dialog dataset on COCO
images \cite{mscoco}, with free-form answers.
The publicly available VisDial v0.9 %
contains $10$-round dialogs on around $83k$ training images, and $40k$ 
validation images.
VisDial was collected from pairs of human workers, by instructing one of them 
to ask questions in a live chat interface to help them imagine the scene 
better.
Thus, the dialogs contain a lot of coreferences in natural language, which need 
to be resolved to answer the questions accurately.

\input{sections/main/visdial_qualitative.tex}

\myparagraph{Models and baselines.}
In addition to the CorefNMN\xspace model described in Sec\onedot\ref{sec:approach}, 
we also consider ablations without the memory network augmented
program generator (CorefNMN\xspace{}$\backslash$Mem) or the auxiliary loss 
$\mathcal{L}_C^{aux}$ to train modules on captions 
(CorefNMN\xspace{}$\backslash\mathcal{L}_C^{aux}$), and without both
(CorefNMN\xspace{}$\backslash$Mem$\backslash\mathcal{L}_C^{aux}$).
As strong baselines, we consider: 
(a) neural module network without history \cite{hu2017learning} with answer
generation,
(b) the best \textit{discriminative} model based on memory networks MN-QIH-D 
from \cite{visdial}, %
(c) history-conditioned image attentive encoder (HCIAE-D-MLE) \cite{lu2017best},
and
(d) Attention-based visual coreference model (AMEM+SEQ-QI) \cite{paul2017visual}.
We use ImageNet pretrained VGG-16 \cite{simonyan_iclr15} to extract $x_{vis}$,
and also ResNet-152 \cite{he16cvpr} for CorefNMN\xspace{}.
Further comparisons are in supplement.

\myparagraph{Evaluation.}
Evaluation in visual dialog is via retrieval of the ground truth answer 
$A^{gt}_t$ from a pool of $100$ candidate answers 
$\mathcal{A}_t = \{A_t^{(1)}, \cdots A_t^{(100)}\}$.
These candidates are ranked based the discriminative decoder scores.
We report Recall$@k$ for $k=\{1, 5, 10\}$, mean rank, and mean reciprocal rank 
(MRR), as suggested by \cite{visdial},
on the set of $40k$ validation images (there is not test available for v0.9).

\myparagraph{Supervision.}
In addition to the ground truth answer $A^{gt}_t$ at each round, our model 
gets program supervision for $Q_t$, to train the program generator.
We automatically obtain (weak) program supervision from a language parser on 
questions (and captions) %
 \cite{hu16cvpr} %
and supervision to predict for \texttt{Refer} from %
an off-the-shelf text coreference resolution
tool\footnote{\url{https://github.com/huggingface/neuralcoref}}, based on
\cite{clark16emnlp}.
For questions that are a part of coreference chain, we replace \texttt{Find}
 with \texttt{Refer} in the parser supervised program. %
Our model predicts everything from the questions at test time. %

\myparagraph{Results.}
We summarize our observations from \Table\ref{tab:visdial_disc_result_table} below:
(a) Our CorefNMN\xspace outperforms all other approaches across all the metrics, 
	highlighting the importance of explicitly resolving coreferences for visual
	dialog.
  	Specifically, our R$@k$ ($k=1,2,5$) is at least 1 point higher than the 	
     best prior work (AMEM+SEQ-QI), and almost 2 points higher than NMN.
(b) Removing memory augmentation (CorefNMN\xspace{}$\backslash$Mem) hurts performance 
	uniformly over all metrics, as the model is unable to peek into 
    history to decide when to resolve coreferences via the \texttt{Refer} 
    module.
	Modules on captions seems to have varied effect on the full model, with
    decrease in R$@1$, but marginal increase or no effect in other metrics.
    %
	%
    %
    %
    %
    %
(c) \Fig\ref{fig:visdial_qualitative} illustrates the interpretable and 	
    grounded nature of our model.
    %

\subsection{VisDial v1.0 Dataset}
\textbf{Dataset.}
Das \textit{et~al\onedot}} \def\iid{i.i.d\onedot \cite{visdial} recently released VisDial v1.0 dataset.
Specifically, VisDial v1.0 comprises of:
(a) A re-organization of train and validation splits from v0.9 to form the new train v1.0.
Thus, train v1.0 now contains $120k$ images with $10-$round dialogs for each images, resulting
in a total of $1.2$ million question-answer pairs.
(b) An additional 10k COCO-like images from Flickr, on which crowd-sourced dialogs between
pairs of humans were collected similar to v0.9.
The 10k images are further split into 2k validation (val v1.0) and 8k test sets (test-std v1.0).
Dense candidate option annotations, which indicate the correctness 
of each candidate in the pool, were also collected for these 10k images.
Each image in the val v1.0 split is associated with a $10-$round dialog, 
while an image in test-std v1.0 has a variable-round dialog.

\noindent
\textbf{Additional Metrics and Models.}
Just as in the previous version (v0.9), the performance on the VisDial v1.0 dataset 
is benchmarked using standard retrieval metrics like Recall$@k$ ($k=\{1, 5, 10\}$),
mean reciprocal rank (MRR) and mean rank.
Further, Das \textit{et~al\onedot}} \def\iid{i.i.d\onedot \cite{visdial} also propose to use normalized 
discounted cumulative gain (NDCG) to score the sorted pool of candidate
answers,
to evaluate on VisDial v1.0\footnote{\url{https://visualdialog.org/challenge/2018}}.
Intuitively, NDCG penalizes accurate answers that appear lower in the 
sorted pool based on a logarithmic weighting scheme, normalizing for 
the number of accurate answers across instances.
We train our CorefNMN\xspace{} model on train v1.0 and report numbers on test-std v1.0 split.
We compare against LF-QIH-D, HRE-QIH-D, and MN-QIH-D from \cite{visdial}, and
out-of-the-box neural module network (NMN) \cite{hu2017learning}.
The LF-QIH-D, HRE-QIH-D, and MN-QIH-D use VGG \cite{simonyan_iclr15} 
image features, while the neural module based models (CorefNMN\xspace{} and NMN) 
use ResNet-152 \cite{he16cvpr} features.

\noindent
\textbf{Results.}

Performance on VisDial v1.0 is given in \reftab{tab:visdial_v1_result_table}.
Our CorefNMN\xspace{} outperforms all other approaches on all metrics, except the neural module
baseline (NMN) on the NDCG metric.
We note that recent state-of-the-art, as reported on the leaderboard\footnote{https://evalai.cloudcv.org/web/challenges/challenge-page/103/leaderboard/298}, 
has reached up to 0.578 (NDCG) from team DL-61, but the approach and 
other details (e.g\onedot} \def\Eg{E.g\onedot. features, use of an ensemble) are not fully known and unpublished at 
this point in time.

\input{sections/main/visdial_v1_result_table.tex}




\subsection{MNIST Dialog Dataset}

Due to the synthetic nature, the text in the dialog is of low variability 
and is made up of a small vocabulary.
Thus, we only use a single-layered LSTM with a hidden size of $64$ to encode 
both questions and history.
For each word in our vocabulary of size $73$, we learn embeddings with $32$
dimensions.
We learn a similar dimensional embedding for each of our modules.
To extract image features, we design a convolutional neural network (CNN) with
the same architecture as \cite{paul2017visual}.
Specifically, our CNN has four $3 \times 3$ convolutional layers, 
each followed by a batch norm, ReLU non-linearity, and a $2 \times 2$ max pool
layer.
While the first two convolutional layers have $32$ feature channels, the last 
two have $64$ channels each.
To pick the best model, we use early stoppage on the provided validation
set of $10k$ images.


\subsection{VisDial Dataset}

Each LSTM used in our VisDial experiments has two layers and with a hidden size
of $1000$.
To represent images, we use convolutional features before the final mean pooling
from a ImageNet pre-trained ResNet-152 \cite{he16cvpr} model.
Further, we also add two additional dimensions indicating the $X$ (columns) and 
$Y$ (rows) locations respectively, to facilitate the model in handling
spatial reasoning.
With a large vocabulary of around $8k$ words, our word and module embeddings 
are $300$ dimensional vectors.
We also initialize our word embeddings with GloVe \cite{pennington2014glove}.
We pick the best model via early stopping using mean
reciprocal rank metric on a subset of $3k$ images, set aside from the
$83k$ training images of VisDial v0.9.

\subsection{Document Changelog}
To help the readers track changes to this document, a brief changelog 
describing the revisions is provided below:

\begin{description}
\item{\textbf{v0:}} ECCV 2018 camera-ready version (not on arXiv).
\item{\textbf{v1:}} Initial arXiv version: Added experiments on VisDial v1.0 dataset.
\end{description}
\vspace*{-10pt}

