  Attention involves comparing query and key vectors in terms of a scalar
product, $\mathbf{Q}^T\mathbf{K}$, together with a subsequent softmax
normalization. Classicaly, parallel/orthogonal/antiparallel queries and keys
lead to large/intermediate/small attention weights. Here we study expressive
attention (EA), which is based on $(\mathbf{Q}^T\mathbf{K})^2$, the squared dot
product. In this case attention is enhanced when query and key are either
parallel or antiparallel, and suppressed for orthogonal configurations. For a
series of autoregressive prediction tasks, we find that EA performs at least as
well as the standard mechanism, dot-product attention (DPA). Increasing task
complexity, EA is observed to outperform DPA with increasing margins, which
also holds for multi-task settings. For a given model size, EA manages to
achieve 100\% performance for a range of complexity levels not accessible to
DPA.
