  Explanation faithfulness of model predictions in natural language processing
is typically evaluated on held-out data from the same temporal distribution as
the training data (i.e. synchronous settings). While model performance often
deteriorates due to temporal variation (i.e. temporal concept drift), it is
currently unknown how explanation faithfulness is impacted when the time span
of the target data is different from the data used to train the model (i.e.
asynchronous settings). For this purpose, we examine the impact of temporal
variation on model explanations extracted by eight feature attribution methods
and three select-then-predict models across six text classification tasks. Our
experiments show that (i)faithfulness is not consistent under temporal
variations across feature attribution methods (e.g. it decreases or increases
depending on the method), with an attention-based method demonstrating the most
robust faithfulness scores across datasets; and (ii) select-then-predict models
are mostly robust in asynchronous settings with only small degradation in
predictive performance. Finally, feature attribution methods show conflicting
behavior when used in FRESH (i.e. a select-and-predict model) and for measuring
sufficiency/comprehensiveness (i.e. as post-hoc methods), suggesting that we
need more robust metrics to evaluate post-hoc explanation faithfulness.
