  This paper considers a network referred to as Modality Shifting Attention
Network (MSAN) for Multimodal Video Question Answering (MVQA) task. MSAN
decomposes the task into two sub-tasks: (1) localization of temporal moment
relevant to the question, and (2) accurate prediction of the answer based on
the localized moment. The modality required for temporal localization may be
different from that for answer prediction, and this ability to shift modality
is essential for performing the task. To this end, MSAN is based on (1) the
moment proposal network (MPN) that attempts to locate the most appropriate
temporal moment from each of the modalities, and also on (2) the heterogeneous
reasoning network (HRN) that predicts the answer using an attention mechanism
on both modalities. MSAN is able to place importance weight on the two
modalities for each sub-task using a component referred to as Modality
Importance Modulation (MIM). Experimental results show that MSAN outperforms
previous state-of-the-art by achieving 71.13\% test accuracy on TVQA benchmark
dataset. Extensive ablation studies and qualitative analysis are conducted to
validate various components of the network.
