  The attention mechanism has been widely used in deep neural networks as a
model component. By now, it has become a critical building block in many
state-of-the-art natural language models. Despite its great success established
empirically, the working mechanism of attention has not been investigated at a
sufficient theoretical depth to date. In this paper, we set up a simple text
classification task and study the dynamics of training a simple attention-based
classification model using gradient descent. In this setting, we show that, for
the discriminative words that the model should attend to, a persisting identity
exists relating its embedding and the inner product of its key and the query.
This allows us to prove that training must converge to attending to the
discriminative words when the attention output is classified by a linear
classifier. Experiments are performed, which validate our theoretical analysis
and provide further insights.
