  When applied to Image-to-text models, interpretability methods often provide
token-by-token explanations namely, they compute a visual explanation for each
token of the generated sequence. Those explanations are expensive to compute
and unable to comprehensively explain the model's output. Therefore, these
models often require some sort of approximation that eventually leads to
misleading explanations. We develop a framework based on SHAP, that allows for
generating comprehensive, meaningful explanations leveraging the meaning
representation of the output sequence as a whole. Moreover, by exploiting
semantic priors in the visual backbone, we extract an arbitrary number of
features that allows the efficient computation of Shapley values on large-scale
models, generating at the same time highly meaningful visual explanations. We
demonstrate that our method generates semantically more expressive explanations
than traditional methods at a lower compute cost and that it can be generalized
over other explainability methods.
