  We provide a probabilistic interpretation of attention and show that the
standard dot-product attention in transformers is a special case of Maximum A
Posteriori (MAP) inference. The proposed approach suggests the use of
Expectation Maximization algorithms for online adaptation of key and value
model parameters. This approach is useful for cases in which external agents,
e.g., annotators, provide inference-time information about the correct values
of some tokens, e.g, the semantic category of some pixels, and we need for this
new information to propagate to other tokens in a principled manner. We
illustrate the approach on an interactive semantic segmentation task in which
annotators and models collaborate online to improve annotation efficiency.
Using standard benchmarks, we observe that key adaptation boosts model
performance ($\sim10\%$ mIoU) in the low feedback regime and value propagation
improves model responsiveness in the high feedback regime. A PyTorch layer
implementation of our probabilistic attention model will be made publicly
available here: https://github.com/apple/ml-probabilistic-attention.
