  Generative transformer models have become increasingly complex, with large
numbers of parameters and the ability to process multiple input modalities.
Current methods for explaining their predictions are resource-intensive. Most
crucially, they require prohibitively large amounts of extra memory, since they
rely on backpropagation which allocates almost twice as much GPU memory as the
forward pass. This makes it difficult, if not impossible, to use them in
production. We present AtMan that provides explanations of generative
transformer models at almost no extra cost. Specifically, AtMan is a
modality-agnostic perturbation method that manipulates the attention mechanisms
of transformers to produce relevance maps for the input with respect to the
output prediction. Instead of using backpropagation, AtMan applies a
parallelizable token-based search method based on cosine similarity
neighborhood in the embedding space. Our exhaustive experiments on text and
image-text benchmarks demonstrate that AtMan outperforms current
state-of-the-art gradient-based methods on several metrics while being
computationally efficient. As such, AtMan is suitable for use in large model
inference deployments.
