  Attention mechanisms are a central property of cognitive systems allowing
them to selectively deploy cognitive resources in a flexible manner. Attention
has been long studied in the neurosciences and there are numerous
phenomenological models that try to capture its core properties. Recently
attentional mechanisms have become a dominating architectural choice of machine
learning and are the central innovation of Transformers. The dominant intuition
and formalism underlying their development has drawn on ideas of keys and
queries in database management systems. In this work, we propose an alternative
Bayesian foundation for attentional mechanisms and show how this unifies
different attentional architectures in machine learning. This formulation
allows to to identify commonality across different attention ML architectures
as well as suggest a bridge to those developed in neuroscience. We hope this
work will guide more sophisticated intuitions into the key properties of
attention architectures and suggest new ones.
