  Deep learning continues to revolutionize an ever-growing number of critical
application areas including healthcare, transportation, finance, and basic
sciences. Despite their increased predictive power, model transparency and
human explainability remain a significant challenge due to the "black box"
nature of modern deep learning models. In many cases the desired balance
between interpretability and performance is predominately task specific.
Human-centric domains such as healthcare necessitate a renewed focus on
understanding how and why these frameworks are arriving at critical and
potentially life-or-death decisions. Given the quantity of research and
empirical successes of deep learning for computer vision, most of the existing
interpretability research has focused on image processing techniques.
Comparatively, less attention has been paid to interpreting deep learning
frameworks using sequential data. Given recent deep learning advancements in
highly sequential domains such as natural language processing and physiological
signal processing, the need for deep sequential explanations is at an all-time
high. In this paper, we review current techniques for interpreting deep
learning techniques involving sequential data, identify similarities to
non-sequential methods, and discuss current limitations and future avenues of
sequential interpretability research.
