  We present VQA-MHUG - a novel 49-participant dataset of multimodal human gaze
on both images and questions during visual question answering (VQA) collected
using a high-speed eye tracker. We use our dataset to analyze the similarity
between human and neural attentive strategies learned by five state-of-the-art
VQA models: Modular Co-Attention Network (MCAN) with either grid or region
features, Pythia, Bilinear Attention Network (BAN), and the Multimodal
Factorized Bilinear Pooling Network (MFB). While prior work has focused on
studying the image modality, our analyses show - for the first time - that for
all models, higher correlation with human attention on text is a significant
predictor of VQA performance. This finding points at a potential for improving
VQA performance and, at the same time, calls for further research on neural
text attention mechanisms and their integration into architectures for vision
and language tasks, including but potentially also beyond VQA.
