  We propose a simple modification to the conventional attention mechanism
applied by Transformers: Instead of quantifying pairwise query-key similarity
with scaled dot-products, we quantify it with the logarithms of scaled
dot-products of exponentials. Our modification linearizes attention with
exponential kernel feature maps, whose corresponding feature function is
infinite dimensional. We show that our modification is expressible as a
composition of log-sums of exponentials, with a latent space of constant size,
enabling application with constant time and space complexity per token. We
implement our modification, verify that it works in practice, and conclude that
it is a promising alternative to conventional attention.
