  Deep learning models developed for time-series associated tasks have become
more widely researched nowadays. However, due to the unintuitive nature of
time-series data, the interpretability problem -- where we understand what is
under the hood of these models -- becomes crucial. The advancement of similar
studies in computer vision has given rise to many post-hoc methods, which can
also shed light on how to explain time-series models. In this paper, we present
a wide range of post-hoc interpretation methods for time-series models based on
backpropagation, perturbation, and approximation. We also want to bring focus
onto inherently interpretable models, a novel category of interpretation where
human-understandable information is designed within the models. Furthermore, we
introduce some common evaluation metrics used for the explanations, and propose
several directions of future researches on the time-series interpretability
problem. As a highlight, our work summarizes not only the well-established
interpretation methods, but also a handful of fairly recent and under-developed
techniques, which we hope to capture their essence and spark future endeavours
to innovate and improvise.
