  With deep neural models increasingly permeating our daily lives comes a need
for transparent and comprehensible explanations of their decision-making.
However, most explanation methods that have been developed so far are not
intuitively understandable for lay users. In contrast, natural language
explanations (NLEs) promise to enable the communication of a model's
decision-making in an easily intelligible way. While current models
successfully generate convincing explanations, it is an open question how well
the NLEs actually represent the reasoning process of the models - a property
called faithfulness. Although the development of metrics to measure
faithfulness is crucial to designing more faithful models, current metrics are
either not applicable to NLEs or are not designed to compare different model
architectures across multiple modalities.
  Building on prior research on faithfulness measures and based on a detailed
rationale, we address this issue by proposing three faithfulness metrics:
Attribution-Similarity, NLE-Sufficiency, and NLE-Comprehensiveness. The
efficacy of the metrics is evaluated on the VQA-X and e-SNLI-VE datasets of the
e-ViL benchmark for vision-language NLE generation by systematically applying
modifications to the performant e-UG model for which we expect changes in the
measured explanation faithfulness. We show on the e-SNLI-VE dataset that the
removal of redundant inputs to the explanation-generation module of e-UG
successively increases the model's faithfulness on the linguistic modality as
measured by Attribution-Similarity. Further, our analysis demonstrates that
NLE-Sufficiency and -Comprehensiveness are not necessarily correlated to
Attribution-Similarity, and we discuss how the two metrics can be utilized to
gain further insights into the explanation generation process.
