  The correct interpretation and understanding of deep learning models are
essential in many applications. Explanatory visual interpretation approaches
for image, and natural language processing allow domain experts to validate and
understand almost any deep learning model. However, they fall short when
generalizing to arbitrary time series, which is inherently less intuitive and
more diverse. Whether a visualization explains valid reasoning or captures the
actual features is difficult to judge. Hence, instead of blind trust, we need
an objective evaluation to obtain trustworthy quality metrics. We propose a
framework of six orthogonal metrics for gradient-, propagation- or
perturbation-based post-hoc visual interpretation methods for time series
classification and segmentation tasks. An experimental study includes popular
neural network architectures for time series and nine visual interpretation
methods. We evaluate the visual interpretation methods with diverse datasets
from the UCR repository and a complex, real-world dataset and study the
influence of standard regularization techniques during training. We show that
none of the methods consistently outperforms others on all metrics, while some
are sometimes ahead. Our insights and recommendations allow experts to choose
suitable visualization techniques for the model and task.
