  Explaining the decision-making process of machine learning models is crucial
for ensuring their reliability and fairness. One popular explanation form
highlights key input features, such as i) tokens (e.g., Shapley Values and
Integrated Gradients), ii) interactions between tokens (e.g., Bivariate Shapley
and Attention-based methods), or iii) interactions between spans of the input
(e.g., Louvain Span Interactions). However, these explanation types have only
been studied in isolation, making it difficult to judge their respective
applicability. To bridge this gap, we propose a unified framework that
facilitates a direct comparison between highlight and interactive explanations
comprised of four diagnostic properties. Through extensive analysis across
these three types of input feature explanations--each utilizing three different
explanation techniques--across two datasets and two models, we reveal that each
explanation type excels in terms of different diagnostic properties. In our
experiments, highlight explanations are the most faithful to a model's
prediction, and interactive explanations provide better utility for learning to
simulate a model's predictions. These insights further highlight the need for
future research to develop combined methods that enhance all diagnostic
properties.
