  Visual attention mechanisms are widely used in multimodal tasks, as visual
question answering (VQA). One drawback of softmax-based attention mechanisms is
that they assign some probability mass to all image regions, regardless of
their adjacency structure and of their relevance to the text. In this paper, to
better link the image structure with the text, we replace the traditional
softmax attention mechanism with two alternative sparsity-promoting
transformations: sparsemax, which is able to select only the relevant regions
(assigning zero weight to the rest), and a newly proposed Total-Variation
Sparse Attention (TVmax), which further encourages the joint selection of
adjacent spatial locations. Experiments in VQA show gains in accuracy as well
as higher similarity to human attention, which suggests better
interpretability.
