  Self-attention (SA) network has shown profound value in image captioning. In
this paper, we improve SA from two aspects to promote the performance of image
captioning. First, we propose Normalized Self-Attention (NSA), a
reparameterization of SA that brings the benefits of normalization inside SA.
While normalization is previously only applied outside SA, we introduce a novel
normalization method and demonstrate that it is both possible and beneficial to
perform it on the hidden activations inside SA. Second, to compensate for the
major limit of Transformer that it fails to model the geometry structure of the
input objects, we propose a class of Geometry-aware Self-Attention (GSA) that
extends SA to explicitly and efficiently consider the relative geometry
relations between the objects in the image. To construct our image captioning
model, we combine the two modules and apply it to the vanilla self-attention
network. We extensively evaluate our proposals on MS-COCO image captioning
dataset and superior results are achieved when comparing to state-of-the-art
approaches. Further experiments on three challenging tasks, i.e. video
captioning, machine translation, and visual question answering, show the
generality of our methods.
