  Interpretable machine learning and explainable artificial intelligence have
become essential in many applications. The trade-off between interpretability
and model performance is the traitor to developing intrinsic and model-agnostic
interpretation methods. Although model explanation approaches have achieved
significant success in vision and natural language domains, explaining time
series remains challenging. The complex pattern in the feature domain, coupled
with the additional temporal dimension, hinders efficient interpretation.
Saliency maps have been applied to interpret time series windows as images.
However, they are not naturally designed for sequential data, thus suffering
various issues.
  This paper extensively analyzes the consistency and robustness of saliency
maps for time series features and temporal attribution. Specifically, we
examine saliency explanations from both perturbation-based and gradient-based
explanation models in a time series classification task. Our experimental
results on five real-world datasets show that they all lack consistent and
robust performances to some extent. By drawing attention to the flawed saliency
explanation models, we motivate to develop consistent and robust explanations
for time series classification.
