  Attention networks show promise for both vision and language tasks, by
emphasizing relationships between constituent elements through weighting
functions. Such elements could be regions in an image output by a region
proposal network, or words in a sentence, represented by word embedding. Thus
far the learning of attention weights has been driven solely by the
minimization of task specific loss functions. We introduce a method for
learning attention weights to better emphasize informative pair-wise relations
between entities. The key component is a novel center-mass cross entropy loss,
which can be applied in conjunction with the task specific ones. We further
introduce a focused attention backbone to learn these attention weights for
general tasks. We demonstrate that the focused supervision leads to improved
attention distribution across meaningful entities, and that it enhances the
representation by aggregating features from them. Our focused attention module
leads to state-of-the-art recovery of relations in a relationship proposal task
and boosts performance for various vision and language tasks.
