  The debate around the interpretability of attention mechanisms is centered on
whether attention scores can be used as a proxy for the relative amounts of
signal carried by sub-components of data. We propose to study the
interpretability of attention in the context of set machine learning, where
each data point is composed of an unordered collection of instances with a
global label. For classical multiple-instance-learning problems and simple
extensions, there is a well-defined "importance" ground truth that can be
leveraged to cast interpretation as a binary classification problem, which we
can quantitatively evaluate. By building synthetic datasets over several data
modalities, we perform a systematic assessment of attention-based
interpretations. We find that attention distributions are indeed often
reflective of the relative importance of individual instances, but that silent
failures happen where a model will have high classification performance but
attention patterns that do not align with expectations. Based on these
observations, we propose to use ensembling to minimize the risk of misleading
attention-based explanations.
